Feature Proposal: TWiki Engines distributed as Contribs

Motivation

Recently, FoswikiStandAloneFeature project was accepted and is about to be merged. However I thought about the possibility to include in core only CGI and CLI engines, since they are fully compatible with actual mechanism and have no additional dependencies.

Other reason are:

• More engines can be developed, besides the ones that already exist
• People that doesn't have checkin rights to core svn can contribute with engines
• Development of engines can be faster (no need to go through core new feature/refactor process)

Description and Documentation

This proposal involves some code that is not yet in core, from FoswikiStandAloneFeature project. That project consists of an abstraction layer, that makes TWiki independent of execution mechanism (TWiki::Request, TWiki::Request::Upload, TWiki::Response and TWiki::LoginManager::Session), an engine infra-structure, that defines a common interface to the abstraction layer iterate with engines (TWiki::Engine), and some engines (CGI, CLI, ModPerl, FastCGI and HTTP standalone).

FoswikiStandAloneFeature proposal was accepted and when I was making the merge plan, I realized that only the abstraction layer, engine infra-structure and CGI+CLI engines really needs to go into core. With them, TWiki will have the same behavior and dependencies that have now. The other engines (FastCGI, ModPerl and HTTP) can be distributed as contribs and this proposal is about it: instead of merging all FoswikiStandAloneFeature project into core, merge only what really needs to go into core. (abstraction layer, engine infra-structure and CGI+CLI engines).

In summary:

Code	Role	Target
TWiki::Request TWiki::Request::Upload TWiki::Response TWiki::LoginManager::Session	Abstraction Layer	Core
TWiki::Engine	Engine Infra-Structure	Core
TWiki::Engine::CGI TWiki::Engine::CLI bin/twiki_cgi.pl	CGI+CLI Engines	Core
TWiki::Engine::FastCGI bin/twiki_fastcgi.pl	FastCGI Engine	Contrib
lib/TWiki/Engine/Apache.pm lib/TWiki/Engine/Apache/MP13.pm lib/TWiki/Engine/Apache2.pm lib/TWiki/Engine/Apache/MP19.pm lib/TWiki/Engine/Apache/MP20.pm	ModPerl Engine	Contrib
lib/TWiki/Engine/HTTP.pm lib/TWiki/Engine/HTTP/Restarter.pm tools/twiki_http.pl	HTTP Engine	Contrib

Each engine consists of a TWiki::Engine subclass, maybe some helper scripts and documentation. They should be named as "FooEngineContrib", currently it would be something like:

• FastCGIEngineContrib
  • lib/TWiki/Engine/FastCGI.pm
  • bin/twiki_fastcgi.pl
  • data/TWiki/FastCGIEngine.txt
• ModPerlEngineContrib
  • lib/TWiki/Engine/Apache.pm
  • lib/TWiki/Engine/Apache/MP13.pm
  • lib/TWiki/Engine/Apache2.pm
  • lib/TWiki/Engine/Apache2/MP19.pm
  • lib/TWiki/Engine/Apache2/MP20.pm
  • data/TWiki/ModPerlEngine.txt
• HTTPEngineContrib
  • lib/TWiki/Engine/HTTP.pm
  • lib/TWiki/Engine/HTTP/Restarter.pm
  • tools/twiki_http.pl
  • data/TWiki/HTTPEngine.txt

And each one would have within its documentation: description, dependencies, how to install, configure and use.

Impact

WhatDoesItAffect:

Install %WHATDOESITAFFECT%

edit

-- Contributors: GilmarSantosJr - 28 Apr 2008

Discussion

On principle I'm very much in favour of modularisation of the core to support different target environments, but I'm not sure exactly what this proposal involves. What parts of the core are you proposing to move into the Engine classes? Can you be more specific? Perhaps there is some code we should be reading?

-- CrawfordCurrie - 28 Apr 2008

I added more details at description above and the contents of contrib package that each engine already developed would have. All the referenced code is at FoswikiStandAloneFeature branch.

-- GilmarSantosJr - 28 Apr 2008

IIUC, what you're proposing is basically to merge the refactoring that extracted the abstractions and the current engines (CGI+CLI), and leave all the engines you implemented as contribs. If that's so, I completely agree with that approach

-- RafaelAlvarez - 28 Apr 2008

Added TWiki::Request::Upload to the the abstraction layer's description above, for completeness.

You got the idea, Rafael

-- GilmarSantosJr - 28 Apr 2008

yes, that sounds like a useful separation - It will make it simpler to do the debian packages too....

What's the timetable for merging it all into trunk?

-- SvenDowideit - 28 Apr 2008

Agreed, that all sounds very sensible. I looked over the code in the branch, and it makes a great deal of sense. A couple of requests, though:

1. Please make sure there are unit tests wherever possible
2. Some more high level overview POD documentation would be nice, especially to help new contributors who may not be familiar with the old code.

Excellent work, Gilmar

-- CrawfordCurrie - 29 Apr 2008

Sven, I'm very busy now and I'm planning to dedicate 6-8 hours/week to play with TWiki. Since it seems to me that we have a consensus about this issue, I'll start to file detailed bug reports and commit changes. I'll try to write very clear reports and to commit all changes in a single turn, since in the middle-of-work trunk will probably be broken.

Thanks, Crawford! I'll consider to write the high-level documentation you suggested. It's very important to anyone that isn't familiar with the code. Unit tests are at my TODO list since a while. I'll work on them as soon as I learn how and have the time for it

-- GilmarSantosJr - 30 Apr 2008

Excellent news Gilmar - Sounds like I should devote a little time to setting up the nightly build and unit tests for the trunk branch, so that we can track your progress I seriously look forward to your work, as it removes alot of the remaining legacies that have held TWiki back.

-- SvenDowideit - 30 Apr 2008

Overall a very nice refactoring work Gilmar!

I am wondering what the performance of standard CGI mode is of TWiki before and after the code refactor. (That is what we ship out of the box.)

-- PeterThoeny - 01 May 2008

I don't have any concerns about this, as long as Gilmar addresses my points above, so I removed my name from the "Concern raised by" field. I note however that Peter is asking for performance assurances.

-- CrawfordCurrie - 26 May 2008

Sounds like we are one simple question from Peter to Gilmar from having consensus on this one.

-- KennethLavrsen - 27 May 2008

I'll take a test by the beginning of June and until that I won't have time to work on this, but after the test TWiki will get top-priority to my free time activities. Then I'll learn how to use tests infrastructure and address Crawford's points.

Peter, for sure there is some overhead: it's the cost to fill a TWiki::Request from %ENV and maybe CGI object, and to get info from TWiki::Response and print to stdout. The benefit of this cost is the independence achieved that makes the whole FoswikiStandAloneFeature possible. Nowadays there is no way to make confident measures, since there are many parameters to be considered and we don't have consensus on some benchmark. Considering simple topics (such as documentation: no SEARCHes, INCLUDEs, etc), the difference is less then 10% (typically 6%. Tests using concurrency level 5 and retrieving the same topic 30 times), as far as I could measure using ab. As the complexity and/or size of topic grows, the difference tends to zero (Other bottlenecks shows up: searches, includes, disk access, etc).

Since I want to work on performance improvements, my next step is to develop some benchmark framework to use as the main guide (to "prove" that some changes really make things faster. Not to guess what parts of code could be improved).

-- GilmarSantosJr - 27 May 2008

Not sure why my name was added in the ConcernRaisedBy field, my question was simply a question, not a concern not to accept this feature. I removed my name, so no concerns by anyone!

-- PeterThoeny - 28 May 2008

It is probably me that added your name Peter. I do this when I believe a question is important enough to stop the 14-day clock.

It seems we have consensus here.

Proposal is accepted.

Target release for this is TWiki 5.0 and this is currently the trunk in SVN.

-- KennethLavrsen - 29 May 2008

Commenting is disabled while not logged in