Feature Proposal: Add Paging To All Distributed Search Topics
Motivation
when running
WebIndex on a too large web, we can DOS the server rendering (and searching for slower search algos)
Description and Documentation
I'd like to add a default pager, and set size to something that grep and render can handle - like 400)
perhaps default with a Web/Site pref?
Examples
Impact
Implementation
--
Contributors: SvenDowideit - 19 Apr 2011
Discussion
+1. Especially on slower VMs under load, something like 400 seems reasonable (add yet-another-pref? :-).
Also while you're at it, I guess you'll be removing use of
VarTOPICLIST in
WebIndex, which would "fix"
Tasks.Item10653 I guess.
--
PaulHarvey - 25 Apr 2011
Disabling
WebIndex all together has always been among the tweaks for xtra lage sites. I am pretty sure a
WebTopicList is still a lot faster than a
WebIndex. So I wonder why we need both.
--
MichaelDaum - 26 Apr 2011
the irony is that
Tasks.Item10653 suggests that for really large sites,
WebIndex is faster than
WebTopicList - where large starts at 50,000 topics in a web and (in my test system, heading past 1,000,000
real topics (not generated, just collected over the years).
whatever happens, I'll be looking into that bug and the end result will probably be that I'll make SEARCH just as fast, and then define the TOPICLIST macro as a SEARCH.
If I can find big enough hardware (not blooming likely atm), I'll import a non-versioned backup of wikipedia into foswiki this year, and we can take it for a spin.
--
SvenDowideit - 26 Apr 2011
Careful: that report is on a mongodb setup exclusively. Assuming a standard search engine, any SEARCH might still be almost insane listing 50k topics .... which will DOS any browser and the user staring at the result anyway. That's why it is good to paginate things as suggested. Still, for all non-mongodb user, it might be good to use TOPICLIST as the fastest alternative to get the list of all topics in a web. If it isn't as fast as a SEARCH for a standard setup, then there's something else going wrong in TOPICLIST that needs fixing. The TOM interface that TOPICLIST uses is pretty straight forward as far as I can see, compared to SEARCH which is a hell lot more complicated.
--
MichaelDaum - 27 Apr 2011
agreed - Whatever (if anything) is done, it must be as fast or faster than the current option.
having
marker
and
selection
in SEARCH won't be a bad thing, and making SEARCH fast if
format
only contains
$web
and
$topic
makes alot of sense.
Interestingly,
TOPICLIST
appears to get most of its aleged speed from the fact it doesn't load the topic, and so doesn't obey ACL's to decide if it should render a topic. This should probably be doccoed.
wrt
MongoDB - the reason its slower than SEARCH is because the Listeners don't get used for many Store API's including getWebList and eachTopic
which I'll rectify soon.
--
SvenDowideit - 28 Apr 2011
I think this was merged and released a long time ago, marking it so.
--
SvenDowideit - 14 Mar 2012