You are here: Foswiki>Tasks Web>Item5704 (23 Sep 2011, GeorgeClark)Edit Attach

Item5704: Performance tuning of HtPasswdUser for large number of users

pencil
Priority: Normal
Current State: No Action Required
Released In: n/a
Target Release: major
Applies To: Engine
Component: HtPasswdUser, Performance
Branches:
Reported By: TWiki:Main.PeterThoeny
Waiting For:
Last Change By: GeorgeClark
As you might now, we have performance issues with the TWiki:Codev/NewTWikiOrgServer2008. A Sun specialist found out that reading the .htpasswd file with 36K entries is very slow. Two suggestion to improve performance:

1. Performance tune reading the data

Here is the patch Sun came up with:
I have modified the program based on a suggestion from a Compiler Expert and the response time with our test has come down to 3.5 sec - a whole second down

here is the diff

--- HtPasswdUser.pm.save1    2008-01-21 19:18:55.000000000 -0800
+++ HtPasswdUser.pm    2008-06-12 10:27:36.510762000 -0700
@@ -121,9 +121,9 @@
   }
   open( IN_FILE, "<$TWiki::cfg{Htpasswd}{FileName}" ) ||
     throw Error::Simple( $TWiki::cfg{Htpasswd}{FileName}.' open failed: '.$! );
-    local $/ = undef;
-    foreach my $line ( split(/\r?\n/, <IN_FILE>) ) {
-        if( $line =~ /^(.*?):(.*?)(?::(.*))?$/ ) {
+    my $line = '';
+    while (defined ($line =<IN_FILE>) ) {
+        if( $line =~ /^(.*):(.*)(::(.*))?/ ) {
           $data->{$1}->{pass} = $2;
           $data->{$1}->{emails} = $3 || '';

He has made more suggestion in rearranging the data structure which gives very good response time based on a microbenchmark that i wrote (attached - the commented ones are the various options i used)

2. Lazy loading of user data

There is no reason to load all users at init time if the user is a guest. The user data should be load late, only when needed, e.g. when an access control check is done, or when a login name needs to be converted to WikiName.

-- TWiki:Main/PeterThoeny - 16 Jun 2008

Or, if you have a large userbase, use a random access file (like DBM) or database backed User Store. TWiki:Plugins.HTTPDUserAdminContrib supports both DBM and database.

The htpasswd data was lazyloaded, but there was a backwards compatibility problem wrt groups or something that forced me to revert that frown, sad smile

marking as urgent and checked in speedup for 4.2.1

-- TWiki:Main.SvenDowideit - 16 Jun 2008

note that the patch as provided by Sun is incorrect and will prevent some users from being able to login - Micha fixed it pretty fast.

-- SvenDowideit - 25 Jun 2008

At the 07 Jul 2008 release meeting it was agreed that this bug has a fix that takes care of the urgency. We keep it open for a more general fix (which requires a rewrite of the user code) but with Normal priority.

-- KennethLavrsen - 09 Jul 2008

no, this bug has not had a fix applied. I incorrectly assumed that the Sun patch worked, and applied it, Micha had to back it out again to be able to login. Therefore, it is still urgent, needs actual analysis, and a proper fix. It is also likely that the reason Sun got a speedup, is because their tested regex is broken.

-- SvenDowideit - 28 Jul 2008

We knew this when we made the decision.

I do not see it realistic to fix this and get it stable in the 5 days we have before the 4.2.1 release.

The alternative - to let people download the buggy 4.2.0 is worse.

I downgrade to normal. Nothing prevents anyone to fix it for 4.2.1 even if the bug is normal graded.

But not releasing 4.2.1 now means this bug will still be in 4.2.0 so it does not resolve anything delaying the 4.2.1 release again.

-- TWiki:Main.KennethLavrsen - 28 Jul 2008

Closing this task. There has been quite a bit of work done on HtPasswdUser and the code in question here is changed. This task is ancient - let's address new issues in a new task.

-- GeorgeClark - 23 Sep 2011

ItemTemplate edit

Summary Performance tuning of HtPasswdUser for large number of users
ReportedBy TWiki:Main.PeterThoeny
Codebase
SVN Range TWiki-5.0.0, Sun, 01 Jun 2008, build 16865
AppliesTo Engine
Component HtPasswdUser, Performance
Priority Normal
CurrentState No Action Required
WaitingFor
Checkins TWikirev:16906 TWikirev:16907 TWikirev:16908 TWikirev:16909
TargetRelease major
ReleasedIn n/a
Topic revision: r14 - 23 Sep 2011, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy