Priority: Normal
Current State: No Action Required
Released In: n/a
Target Release: major
Applies To: Engine
Component: HtPasswdUser, Performance
Branches:
As you might now, we have performance issues with the
TWiki:Codev/NewTWikiOrgServer2008. A Sun specialist found out that reading the .htpasswd file with 36K entries is very slow. Two suggestion to improve performance:
1. Performance tune reading the data
Here is the patch Sun came up with:
I have modified the program based on a suggestion from a Compiler Expert
and the response time with our test has come down to 3.5 sec - a whole second down
here is the diff
--- HtPasswdUser.pm.save1 2008-01-21 19:18:55.000000000 -0800
+++ HtPasswdUser.pm 2008-06-12 10:27:36.510762000 -0700
@@ -121,9 +121,9 @@
}
open( IN_FILE, "<$TWiki::cfg{Htpasswd}{FileName}" ) ||
throw Error::Simple( $TWiki::cfg{Htpasswd}{FileName}.' open failed: '.$! );
- local $/ = undef;
- foreach my $line ( split(/\r?\n/, <IN_FILE>) ) {
- if( $line =~ /^(.*?):(.*?)(?::(.*))?$/ ) {
+ my $line = '';
+ while (defined ($line =<IN_FILE>) ) {
+ if( $line =~ /^(.*):(.*)(::(.*))?/ ) {
$data->{$1}->{pass} = $2;
$data->{$1}->{emails} = $3 || '';
He has made more suggestion in rearranging the data structure which gives
very good response time based on a microbenchmark that i wrote (attached - the
commented ones are the various options i used)
2. Lazy loading of user data
There is no reason to load all users at init time if the user is a guest. The user data should be load late, only when needed, e.g. when an access control check is done, or when a login name needs to be converted to WikiName.
--
TWiki:Main/PeterThoeny - 16 Jun 2008
Or, if you have a large userbase, use a random access file (like DBM) or database backed User Store.
TWiki:Plugins.HTTPDUserAdminContrib supports both DBM and database.
The htpasswd data
was lazyloaded, but there was a backwards compatibility problem wrt groups or something that forced me to revert that
marking as urgent and checked in speedup for 4.2.1
--
TWiki:Main.SvenDowideit - 16 Jun 2008
note that the patch as provided by Sun is incorrect and will prevent some users from being able to login - Micha fixed it pretty fast.
--
SvenDowideit - 25 Jun 2008
At the 07 Jul 2008 release meeting it was agreed that this bug has a fix that takes care of the urgency. We keep it open for a more general fix (which requires a rewrite of the user code) but with Normal priority.
--
KennethLavrsen - 09 Jul 2008
no, this bug has not had a fix applied. I incorrectly assumed that the Sun patch worked, and applied it, Micha had to back it out again to be able to login. Therefore, it is still urgent, needs actual analysis, and a proper fix. It is also likely that the reason Sun got a speedup, is because their tested regex is broken.
--
SvenDowideit - 28 Jul 2008
We knew this when we made the decision.
I do not see it realistic to fix this and get it stable in the 5 days we have before the 4.2.1 release.
The alternative - to let people download the buggy 4.2.0 is worse.
I downgrade to normal. Nothing prevents anyone to fix it for 4.2.1 even if the bug is normal graded.
But not releasing 4.2.1 now means this bug will still be in 4.2.0 so it does not resolve anything delaying the 4.2.1 release again.
--
TWiki:Main.KennethLavrsen - 28 Jul 2008
Closing this task. There has been quite a bit of work done on
HtPasswdUser
and the code in question here is changed. This task is ancient - let's address new issues in a new task.
--
GeorgeClark - 23 Sep 2011