You are here: Foswiki>Tasks Web>Item1363 (28 Sep 2009, AndrewJones)Edit Attach

Item1363: port and enhance KinoSearch

pencil
Priority: Normal
Current State: Closed
Released In:
Target Release: n/a
Applies To: Extension
Component: KinoSearchContrib
Branches:
Reported By: WillNorris
Waiting For: Main.AndrewJones
Last Change By: AndrewJones
wbniv@r17311:~/foswiki.org/trunk$ core/tools/twiki2foswiki SearchEngineKinoSearchAddOn
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/System
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/SearchEngineKinoSearchPlugin.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/KinoSearch.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/SearchEngineKinoSearchAddOn.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/VarKINOSEARCH.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Stringifier.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/KinoSearch.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Index.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifyBase.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/DEPENDENCIES
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/MANIFEST
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/HTML.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/Text.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/XLS.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/PDF.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_abiword.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_antiword.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_wv.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/PPT.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/build.pl
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Config.spec
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Plugins/SearchEngineKinoSearchPlugin.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Plugins
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store/SearchAlgorithms/Kino.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store/SearchAlgorithms
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/System
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn/KinoSearchResult.jpg
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn/KinoSEARCH.jpg
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki
core/tools/twiki2foswiki: line 76: cd: /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Plugins/SearchEngineKinoSearchAddOn: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read build.pl: No such file or directory
sed: can't read build.pl: No such file or directory
sed: can't read DEPENDENCIES: No such file or directory
-- WillNorris - 24 Mar 2009


rewrote tests from using {session}->{store} to Foswiki::Func calls

GETTING CLOSER!!!

current state:

wbniv@r17311:~/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn$ perl build.pl test
Loaded config from /home/wbniv/.buildcontrib
Running tests in /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm
Running: perl -w -I /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /etc/perl -I /usr/local/lib/perl/5.8.8 -I /usr/local/share/perl/5.8.8 -I /usr/lib/perl5 -I /usr/share/perl5 -I /usr/lib/perl/5.8 -I /usr/share/perl/5.8 -I /usr/local/lib/site_perl -I . /home/wbniv/foswiki.org/trunk/core/test/bin/TestRunner.pl /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm
Options:
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Update started
Updating index finished
Update started
Updating index finished
Update started
Updating index finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Use of uninitialized value in split at SearchTests.pm line 119.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 387.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 388.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 403.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 418.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 422.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 423.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 424.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 425.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 426.
Failed to perl -w -I /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /etc/perl -I /usr/local/lib/perl/5.8.8 -I /usr/local/share/perl/5.8.8 -I /usr/lib/perl5 -I /usr/share/perl5 -I /usr/lib/perl/5.8 -I /usr/share/perl/5.8 -I /usr/local/lib/site_perl -I . /home/wbniv/foswiki.org/trunk/core/test/bin/TestRunner.pl /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm: 11 at /home/wbniv/foswiki.org/trunk/core/lib//Foswiki/Contrib/Build.pm line 820, <DATA> line 1.

-- WillNorris - 2 Jun 2009

We might also want to take some of the enhancements from TWiki.org (see http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item6177).

We can certainly take the extra stringifier's from http://develop.twiki.org/svn/twiki/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/. The other enhancements might need reviewing, as some of them either don't make sense or don't seem to be implemented in the best way.

I am quite happy to help out on this (and on Item8246), but I don't want to touch anything without checking with you first.

-- AndrewJones - 07 Sep 2009

Task list for porting and enhancing Kino:

  • Get working under 01x00 and trunk
  • Strinigifiers and tests for .docx, .xlsx, .pptx
  • Try and get unit tests to work (need to learn a lot more about unit testing Foswiki)
  • Move program paths into Configure (so we don't need to assume antiword, etc is in the path)
  • Use Foswiki::Sandbox rather than =system=
  • Improve the docs and default templates
  • Check under FastGGI
  • Integrating KinoSearch into Foswiki's internals =SEARCH=

Other tasks for the future, but might not be completed before the initial port
  • Use rest handlers
    • Might have to wait until Foswiki 1.1, as in 1.0 rest handlers need to be authenticated (afaik)
  • Remove the =%SPLIT%='s from templates

-- AndrewJones - 09 Sep 2009

Using KinoSearch as a SearchEngine might get problematic, since Kinosearch is - by design - not a fulltext search engine: ie. you simply cannot find substrings in tokens.

-- OliverKrueger - 10 Sep 2009

Yeah true, but it could fall back to the default search as needed (i.e. when using regex search).

It looks like a work in progress by SvenDowideit so I will leave it well alone for now.

-- AndrewJones - 10 Sep 2009

Still can't get the unit tests to work, but apart from that this is pretty much ready. Will do a bit more testing and try some more with the tests, but this should be released by the end of the week

-- AndrewJones - 14 Sep 2009

All tests passed (86)

smile

-- AndrewJones - 24 Sep 2009

After 64 checkins, the Kino Search is now ported as KinoSearchContrib and KinoSearchPlugin.

I posted an announcement on the Foswiki blog.

Closing the task.

-- AndrewJones - 24 Sep 2009

Removed old SearchEngineKinoSearchAddOn from svn.

-- AndrewJones - 28 Sep 2009

ItemTemplate edit

Summary port and enhance KinoSearch
ReportedBy WillNorris
Codebase
SVN Range Foswiki-1.0.0, Thu, 08 Jan 2009, build 1878
AppliesTo Extension
Component KinoSearchContrib
Priority Normal
CurrentState Closed
WaitingFor AndrewJones
Checkins Rev 3255 not found Rev 3256 not found Rev 3257 not found Rev 3258 not found Rev 3259 not found Rev 3260 not found Rev 3284 not found Rev 3303 not found Rev 3304 not found Rev 3305 not found Rev 4013 not found Rev 4014 not found Rev 4015 not found Rev 4019 not found Rev 4020 not found Rev 4827 not found Rev 4828 not found Rev 4829 not found Rev 4835 not found Rev 4836 not found Rev 4837 not found Rev 4838 not found Rev 4901 not found Rev 4902 not found Rev 4913 not found Rev 4914 not found Rev 4915 not found Rev 4923 not found Rev 4935 not found Rev 4938 not found Rev 4962 not found Rev 5070 not found Rev 5071 not found Rev 5072 not found Rev 5073 not found Rev 5074 not found Rev 5077 not found Rev 5096 not found Rev 5098 not found Rev 5102 not found Rev 5103 not found Rev 5113 not found Rev 5114 not found Rev 5115 not found Rev 5116 not found Rev 5117 not found Rev 5118 not found Rev 5121 not found Rev 5129 not found Rev 5131 not found KinoSearchContrib:f9ff0c937676 KinoSearchPlugin:e0a9722b800c KinoSearchContrib:88083142465b KinoSearchContrib:4feb7c059b52 KinoSearchContrib:216f1122176a KinoSearchContrib:b2137cbe4514 KinoSearchContrib:4047dfc9238b KinoSearchPlugin:ee15f5c7b57c KinoSearchContrib:e1f5f0eca5a6 KinoSearchPlugin:98249c4dffbf KinoSearchContrib:f6391929f8f0 Rev 5162 not found
TargetRelease n/a
ReleasedIn
Topic revision: r79 - 28 Sep 2009, AndrewJones
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy