Item1646: Trunk only: search uses lots of memory

A search with topic scope that uses a wildcard causes the view script to consume vast amounts of memory. I killed the view script before it finished. Here is a simple example:

%SEARCH{scope="topic" "Bananana*"}%

Wildcard searches with the default scope are fine. Non-wildcard searches with topic scope are also fine.

This bug does not apply to the release branch - only to trunk.

-- MichaelTempest - 20 May 2009

[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] | Count  |  Time (Min/Max) | Memory(Min/Max) | Total      | Method |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |     16 /     16 | 0.000354052 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_getTopicList |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      2 |  0.000 /  0.000 |      0 /      0 | 2.09808e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_makeTopicPattern |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 7.00951e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_searchTopics |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 2.19345e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_tokensFromSearchString |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 1.28746e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::finish |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      8 /      8 | 2.81334e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::new |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] |      1 |  0.028 /  0.028 |    124 /    124 | 0.0278509 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU) | Foswiki::Search::searchWeb |
[Thu May 21 17:27:58 2009] [error] [client 127.0.1.1] 



[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] SEARCH Bananana* took 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 0.843335 wallclock secs ( 0.51 usr +  0.06 sys =  0.57 CPU)
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] | Count  |  Time (Min/Max) | Memory(Min/Max) | Total      | Method |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 0.00010705 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::InfoCache::new |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 4.91142e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_getListOfWebs |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |     16 /     16 | 0.000431061 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::_getTopicList |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 1.38283e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::finish |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.001 /  0.001 |      0 /      0 | 0.00135493 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::formatResults |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      8 /      8 | 2.38419e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::new |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.821 /  0.821 |  29144 /  29144 | 0.82063 wallclock secs ( 0.50 usr +  0.06 sys =  0.56 CPU) | Foswiki::Search::searchWeb |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] |      1 |  0.000 /  0.000 |      0 /      0 | 5.19753e-05 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU) | Foswiki::Search::sortResults |
[Thu May 21 17:28:20 2009] [error] [client 127.0.1.1] 

wow.

mmmm, tracked it down using the new Monitor code

its worrying

its the

                # FIXME I18N
                $qtoken = quotemeta($qtoken) if ( $options->{'type'} ne 'regex' );

line in (for eg) package Foswiki::Store::SearchAlgorithms::Forking;

comment it out, and it goes back to small..

I'm doing more prodding

-- SvenDowideit - 21 May 2009

ok, so this idiot was calling quotemeta on the same string a random number of times - very dumb. but how do we know its not been called elsewhere? Doing so didn't kill any tests :/

fixated.

-- SvenDowideit - 21 May 2009

interestingly, the example that found the issue is a non search, as wildcards don't work unless you set to type='regex', which avoids the problem..

and so we've squashed an unusual corner case - thanks