Feature Proposal: Foswiki dependencies and installation simplicity.

Motivation

This is an attempt to find a way to resolve some matters of dependencies installation.

Description and Documentation

We need user base. The user base needs a wide open entrance to Foswiki world. Lets see what we can do about the CPAN hell of Foswiki dependencies, freeing developers from packaging jail and few more subjects.

The slogan

Free the developer, save the user!

(I'll try to make it some fun here too.)

Preface

Questions are raised about Foswiki dependencies installation. One of the most desperate of them has been seen recently on IRC. The quote:

For WordPress, I install/configure Nginx, and MySQL. Then I drop in the WordPress files. Done. Why is FosWiki so much more complicated?

Though generally the question was more about configuring Apache but it contains the real demand of a decent admin (and I would call him 'a user' because he is using what we're developing): "Do it in a way that I won't have to bother myself with gory details of low-level setup. Let me just unpack it and let it go". Really, who would ever bother looking for tools/dependencies_installer.pl script? And what kind of a disappointment is it when one would find out that the script doesn't work on FreeBSD or rpm-based distro? Neither the script saves in a case of hosted environment with no shell available.

CPAN shell is the universal tool but it demands a user to do manual work of searching/installing the dependencies; it could also turn one's local Perl lib/ subdir into a hard to resolve mess especially for cases when packages install into the same directory.

Packages are to cause another headache when a distro has no package for a popular module. Why? Because the authors decided so. Is there any chance to find out why CentOS doesn't have rpm for Moose module? Doubtfully.

Where does it get us? Into a situation where a inexperienced admin is scared with Perl's CPAN hell. Where a developer is reinventing a wheel because the existing Wheel module is disliked by a distro creator and it is easier to include Foswiki's own implementation of this functionality rather than make an kenneth-admin (any similarity with real people is accidental) unhappy about manual module installations.

Solution?

Not really. An approach allowing to get closer to it. What would be required to resolve the issues of the current situation? The best answer is: _"Use the module builder, Luke!"_ Well, it must be 'a' not 'the' but the classics demands...

Of course, the best doesn't mean it resolves everything but at least this would be the first step to have Foswiki prepared for a packaging system of almost any kind – would it be a Linux beloved one or a BSD-based; not relevant as soon as Foswiki can generate a working setup with a two commands of ExtUtils::MakeMaker or Module::Build.

But that's another big project somebody must be brave enough to get into his workaholic hands. Meanwhile we'd need a workaround. It must fulfill the following requirements:

1. Install required dependencies without any user interaction. A skilled admin would install them explicitly if he wants to. An unskilled won't bother himself with this.
2. Don't mess up with system libraries. Use local source of modules. Let the installation owner to clean everything up without being afraid of consequences.
3. Don't override the existing modules with locally installed unless specifically requested. This is a minor requirement for a intranet installation but pretty important for a world-exposed one.
4. Support limited (jailed) hosted environments where pretty much everything is disabled.

The solution would still be a palliative but combined with PSGI where not that much of pre-configuration is required could serve as a starting point to launch Foswiki with least possible efforts applied.

So far the following commands did launch Foswiki for me on a empty Linux Mint installation:

$ cd $FOSWIKI_HOME; ./pseudo_install.pl developer
$ plackup bin/foswiki_debug.pm

That took some time to deploy the missing modules but eventually it was possible to load working configure page.

Possible issues.

Mostly network timeouts. Perhaps caused by virtualized environment they resulted in failed installations and in pretty much familiar 'white screen of Perl errors' consequently.

Security matters.

Well, an above mentioned kenneth-admin would also raise a question of possible security problems with untested modules and that only selection variety of modules from a magically-protected repo must be allowed or the whole human race is doomed.

While being true in general that worry must also be viewed from another angle: Foswiki has a pretty high entry threshold. For some it might be defined as 'insane'. Just look at the quote above: WordPress, the big and the ugly, is obedient as a trained puppy. And I'm not talking here about some PHP idiotic problems which are not solvable without StackOverflow collective wisdom or professional expertise (Perl has his own concerns after all); they pop up later when one starts utilizing the systems at its full power. But before that it is as simple as to unpack the code in /somewhere/www/wordpress/ and command a web server: "Fetch it!" At this point nobody thinks of security. It's only the first impression which does matter.

Now, as an admin finished playing with his test setup and (hopefully) thinking of deploying the system into production he becomes concerned about the security. What options does he has?

1. Secure packages as mentioned already.
2. Patched versions served through a local CPAN mirror – mostly for big companies capable of spending few days on setting up one.
3. For others – carefully selected versions of published modules. For example, if we know of problems with SomeModule in version 1.10 we can specifically define 1.09 to be installed is it was known to be the most secure version of the module.

Manual installation/patching of modules is not considered as unacceptable for a decent admin specimen and something always possible for a enthusiastic professional. Though I would still prefer local CPAN mirror. Similar by approach though way more universal solution.

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

Foswiki::Aux::Dependencies module on GitHub.

-- Contributors: VadimBelman - 28 Oct 2016

Discussion

We don't ship tools/dependencies_installer.pl, so no sense looking for it. It depends on CPANPlus which is deprecated, and as you mention only supports debian for OS packages.

I'm really uneasy about foswiki doing anything automatically in this area. There have been several attempts, and all seem to end up with holes in feet. At one time way-back-when, the shell version of the extensions installer (implemented as part of extender.pl) also tried to automate CPAN module installations. Then there was the tools/dependencies_installer.pl which we shipped, and then dropped due to issues.

I have great trepidation about all of this.

By the way, the same user who made the "why is this so difficult" comment, also asked why we don't have a docker container. There are actually are a couple of them, created by SvenDowideit, which are unfortunately dependent upon the obsolete debian/rpm packages that were never updated for Foswiki 2.x.

Maybe Docker might be a way to resolve this issue without any need to complicate foswiki internals.

-- GeorgeClark - 28 Oct 2016

See also:

• FoswikiVagrant
• MoveCPANLibToSeparateContrib
• ScanForMissingPerlModules (The genesis of dependencies_installer.pl)
• Tasks.Item13519 - Related - avoid crash for missing dependencies.
• Tasks.Item13266 - Task that removed tools/dependencies_installer.pl
• Tasks.Item13263 - Strategy discussion on perl dependencies

-- GeorgeClark - 28 Oct 2016

Have a look at other communities: python, javascript, php, you-name-it. They have the same problem and thus come with an installer and dependency infrastructure of their own.

And perl has got it too, as long as your perl packages are on CPAN of course.

For me the logical consequence is to move Foswiki and plugins to CPAN, not to invent yet another installer.

Testing out Foswiki should be as simple as:

cpanm Foswiki
plackup bin/foswiki.psgi

-- MichaelDaum - 30 Oct 2016

While me (plus/minus periodically) checking the IRC-logs and the Dev-web, usually do not commenting... But now simply I must. This sentence:

For me the logical consequence is to move Foswiki and plugins to CPAN, not to invent yet another installer.

should be gold framed and put as sticky to the every page in the trunk for the devs. YES, YES YES!

Also

• the CPAN shell ( the best is the mentioned cpanm ) correctly installs all needed dependencies without manual intervention
• the CPAN:cpanfile and CPAN:Carmel provides a great help for the developers
• the CPAN:Carton simplyfies the end-user packaging

About AddDockerContainerForDistributionMethod : it is great idea to use Docker for the perl itself. Why? Because the for the every one Docker the perl is compiled on the target system, e.g. when you installing the Docker container for the perl, it simply downloads the perl sources and compiles it in the target system, and installs cpanm by default. Curious? Just check the content of the Dockerfile at the https://hub.docker.com/_/perl/ . e.g.: https://github.com/perl/docker-perl/blob/7e333260aac375a11f63d632eec38db2454ace7e/5.022.002-64bit/Dockerfile

So, combination of the Docker image (compiles perl + installs cpanm) and CPAN-ified Foswiki codebase could result an really one-line Foswiki installation.

But also, need to tell: All of the above makes sense only in Foswiki 4.0 e.g. when the codebase will "cleaned enough" with "well-separated" responsibilies. Probably the 3.0 will not be ready enough.

-- JozefMojzis - 31 Oct 2016

I have been looking into this area quite a bit lately, see FoswikiVagrant and FoswikiVagrant#Full_Dependency_Management in particular.

In terms of dependency management I was considering my own code to more completely manage Foswiki dependencies, but that in itself is a lot of work.

I am indeed moving towards using CPAN because:

1. This already has substantial dependency management
2. It already has substantial tools to help build and install any module
   • Including external and compiled packages
3. It does not block further virtualized or containerized Foswiki
   • Indeed it serves as a stepping stone towards these goals
   • Or towards Ansible/Chef/Puppet
4. It's a logical choice for a perl project
5. It would help me to be more focused
   • i.e. I'm generating more Qs than As in the various things I'm working on and it's hard to know where to go next

With respect to CPAN, certain decisions will still need to be made

1. Where to install the Foswiki code
   • /usr/lib/foswiki etc (ermm shouldn't that be /etc :; )
   • Probably varies between distro — the point is that it's a known fixed location
2. That is to say only one Foswiki codebase per base-machine
   • This is solved by having more than one base-machine (using VM or container to extend one physical machine — even plenv as a containerization of sorts)
3. Can/should the install update apache/nginx configuration or should that be in separate Foswiki::Nginx, Foswiki::Apache distributions
4. Sensible defaults are OK for a virgin install

JozefMojzis mentioned "well-separated" responsibilities, well some are more important than others. Dependencies between code in the Foswiki core will often not impact installation concerns. For example Foswiki::Time is unnecessarily dependent of $Foswiki::cfg variables, but is not an installation issues. However Foswiki::Store certainly is very important.

Foswiki::Store has been extracted out with a clean API (although getWorkArea shouldn't be in there). It should be possible to upgrade a Store separately from Foswiki itself and mostly it is. The difficulty is the mixing of project code and data kept in the Foswiki store with user code and data kept in the Foswiki Store (separate to Foswiki/user code managed outside the Store). Over time we have separated these things more and more but that work is still incomplete.

My next task is to prepare this project: https://github.com/Jlevens/Win32_Links to be up-loadable to PAUSE that will give me some CPAN experience.

Random thoughts:

• Should FoswikiX::PlainFileStoreContrib (or other Stores) be installed separately to Foswiki::TopicData
  • How to load the relevant Store (if applicable) with this TopicData as part of the install
• How to update LSC within the cpan install process
• Foswiki::Bundle::Vanilla - minimal install, assume Nginx?
• Foswiki::Bundle::FullFat - all extensions and the kitchen sink

So much to say ...

But now I need to go home

-- JulianLevens - 01 Nov 2016

I would like to leave here a more precise explanation from the last release meeting: the point of this proposal is to simplify entry for a newcomer. Especially a inexperienced newcomer.

It is mentioned in the proposal that use of a module builder (like ExtUtils::MakeMaker of Module::Build) is the way to go. And this is the way to CPAN. But until somebody is brave enough to take that way we're stuck where it is all now. This won't change soon because the amount of work to utilize a standard builder is no less than the amount I spent converting the core for Moo. Not until somebody wake up some day with a thought that he's crazy enough for this job.

Meanwhile who would guess how many people dropped the idea of using Foswiki after the initial disappointments? Get apache configured properly, get the dependencies installed and check that everything is in place... I can hardly count the number of possibly useful projects dumped into the virtual trashcan after an hour or two of attempts to get it just compiled or started.

This is the cornerstone of this proposal. I wouldn't bother myself would it still be CGI-Apache based where dependencies installation is the least problem of all. But with PSGI it does make sense even though the first time it would take insanely long time for the script to start.

The second priority is limited hosting environments block their users from manual module installation. Unfortunately, we're helpless with XS-only dependencies but then anyway hosts not delivering these modules for the user are useless and there is nothing we can do about it (except for a remote cross-platform compiling on demand; anybody to implement it and guarantee it's robustness?). But that'll be working solution if the problem is about Perl-only modules one cannot install manually.

Any other use of this proposal is nothing but a side effect of the implementation aimed at the primary targets. Though it won't be a big deal to get it supporting private CPAN-mirrors (where only secure versions of the deps could be kept), do upgrades of the installed modules, etc.

-- VadimBelman - 01 Nov 2016

This is a better direction to consider: http://snapcraft.io/

-- JulianLevens - 02 Nov 2016

I am committed to finding a really good way for us to package up Foswiki for:

1. First time users to have 'one-click' installs
   • Critical to gain new users (as you say people give up after an hour or so)
2. Easy upgrades
3. Enterprise installs and upgrades
   • Git based - I'm doing this with the Foswiki at work which means:
     • git pull
     • sudo service foswiki restart
     • That's all folks — it works here
       • A complete upgrade requires that I need to repeat that for all local extensions
       • Some upgrades will require more work
         • cpan modules
         • web-server config changes
         • LSC changes
         • Tools to fix-up something (e.g. recent ACL changes; unicode upgrade)
       • But all of this can be automated
         • Will require changes to Foswiki to separate out relevant concerns
         • pseudo*-install needs to be replaced by a plain *install (still symlinks and git but further work required to become a full install)

More to come ...

-- JulianLevens - 02 Nov 2016

Just for the note: The world isn'nt "Linux-only". Need to find an installation method which will "works and fit" to: FreeBSD, macOS, Linuxes and probably for the Windows-8/10+ too.

-- JozefMojzis - 02 Nov 2016

+1. Snap packages while they sound good, really are not as cross-platform as might be suggested by their marketing. Some of the "endorsements" of the package format have been questionable.

As far as git based pull & go, that omits all of the compression done as part of our build process. Unless we intend to not support compressed/obfuscated css / js, git pull & go is not really ready for mainstream.

We have to be very careful about what we land on for packaging, that not only is it well supported cross-platform, but that the project team has the tools and developers to be able to maintain the format in perpetuity. We have old packages. Volunteers have built packaging for the debian (deb) and redhat (rpm) ecosystems, as well as Windows (msi), and OSx, none of which are currently maintained. And they used tools that were not "open", so there was no way for the project to continue the path once the volunteers moved on.

So there are two major areas:

• The "core" package.
  • Completely platform agnostic.
  • Automated by our "build" scripts (BuildContrib)
  • Open / free tools. Dev's must not have to purchase software.
• Convenience packages
  • May be distribution specific (msi, rpm, deb, snap, ...)
  • Must by build-able by anyone in the project.
  • Well documented and automated by BuildContrib (or it's successor)
  • Must not be dependent upon commercial $oftware

And none of these should require network access. Some installations block external access by the server.

-- GeorgeClark - 02 Nov 2016

George, I'm not sure we can avoid network access as long as we need dependencies. I'm not happy about limiting a network application with access to the global network. I'm not happy about taking care of one or two cases to the detriment of hundreds of others.

Actually, I once had a employer with limitations like this – quit in a couple of months and got myself another job.

BuildContrib, I'm afraid, is bad when it comes to CPANification of Foswiki. It's just not compatible and must be replaced with something standard.

And, finally, I demand more freedom on choosing what modules to use. Foswiki is constantly re-invent a wheel only to remain as self-contained as possible. Anybody to calculate how many hours/days/weeks/moths were wasted on this?

Since the discussion drifted into the packaging waters my opinion would be:

• no third party packaging tools until core makes its way into CPAN. It means: no self-developed build frameworks;
• standardization of test framework (TAP compliance would be sufficient);
• separation of code and data on FS level is desirable but not mandatory;
• immutable system webs to reduce give a chance to unattended automated upgrades;
• a module similar to perl's standard Config.pm is must for third-party extensions to know where to install their code and how to configure themselves to comply with active Foswiki setup;

When those requirements are fulfilled people would start creating packages for the project on their own, without any effort from the development team. We don't have to provide a solution – we have to provide the tools.

-- VadimBelman - 02 Nov 2016

Vadim, the sites that block network access know how to deal with it, be it by using a local CPAN mirror, or manually copying in tarballs. Painful, but not our problem. The current extension installer handles it just fine by looking for the tarball in the home directory. However if we just crash or refuse to install without network access, that would be a huge problem. Yes these are corner cases, but they are also often the largest customers.

If you are saying your changes will absolutely require internet access for a successful foswiki installation, then I'd raise a formal concern to the proposal. Make it easy for the site with network access I wholeheartedly support. But if you are telling customers that they can't install foswiki without access to the internet, then IMHO that's a complete non-starter.

-- GeorgeClark - 02 Nov 2016

I'll put it other way around: agree, blocking is no good. I didn't think of it this way while was writing the comment but the module could be extended to support this kind of functionality too. Actually, even the current half-demo functionality doesn't fail on errors but only records them for later processing.

What I would strongly disagree with is if a corner case would dictate us the rules.

-- VadimBelman - 02 Nov 2016

I agree with Vadim.

The whole point is to ease software installation. If network access is blocked, things aren't easy anyway, not only for Foswiki. One can't expect it to still be easy.

And we are not alone in requiring network access to install a software package. You find this quite often in other projects, in standard software, but also for open source communities that are highly componentized, such as javascript or python or php.

Perl has got this kind of infrastructure in place as well ... for a very long time already.

-- MichaelDaum - 03 Nov 2016

One more try. There are at least two cases where this is NOT the way forward.

1. Large corporation that requires all code be taken from distribution "supported" repositories. Makes exception for Foswiki, but not all of cpan.
2. Any site installing on CentOS, RHEL, or other distributions that enable Role based security, and block the web server from making outbound http connections.
3. A small number of sites that use proxy firewalls for secured external access, or block all access. (Most likely a requirement in businesses following Credit Card and other highly secured networks).

And my objection is NOT that we are installing using cpanm. It that we are saying that everyone must accept this process. That there is no way to turn it off and still make a successful foswiki installation. That is a HUGE change and I raise my concern. I'll drop the concern if there is some way to disable auto-installation of external 3rd party code.

-- GeorgeClark - 03 Nov 2016

George, well understood.

The argument goes like this:

1. If you are looking for an easy way to install Foswiki, then using cpanm is the best strategy for the project, imho.
2. If you don't have access to the internet during the install procedure, then you will have to download each package yourself and unzip each one by one, worst case. This by definition is not easy and requires additional admin skills.

I don't see why striving for (1) is blocking (2).

-- MichaelDaum - 03 Nov 2016

Nor do I ... I'm in agreement. However in some IRC discussions, the code has no way to disable the feature. If a dependency is missing, it gets installed, and when the install runs, if it fails, foswiki fails.

What I'm requesting is

1. that a site that does NOT want to use autoinstalled cpan, or cannot use it due to blocked network access can still use foswiki.
2. That there is some way to turn off autoinstall, for sites where there are administrative restrictions on unapproved code.
3. That foswiki tell you in a relatively graceful way that you are missing dependencies, as it does today. A "500" internal error ... go read the logs ... is not acceptable. It needs to present a "You are missing Blah..." when possible, not "just install it or croak"

So maybe I'm misunderstanding the proposal. Just give me a way to turn off the auto-installer. If I want or am required to use the .deb / .rpm packages for code, that should be my choice. How about "touch lib/NOCPAN" and core forgets about the magic. An easy way to disable this feature for those who do not want it, that will persist across upgrades.

-- GeorgeClark - 03 Nov 2016

And one other comment to Vadim's "And, finally, I demand more freedom on choosing what modules to use. Foswiki is constantly re-invent a wheel only to remain as self-contained as possible."

There is a reason to choose well supported CPAN (which to me says that the major distributions support it). As I watch servers auto-update "Security" patches every night, it's reassuring to know that the CPAN modules are part of it. Is this appropriate for everyone, no. And are we saying that you cannot use CPAN, NO again. Only that the project use good judgement in choosing which CPAN modules to use, and when there are alternatives, stick to modules well supported by the distros.

Foswiki is first and foremost an "enterprise wiki" ... And it is reasonable that we strive to use modules supported by the "enterprise distributions".

Yes it's frustrating as hell when distros don't pick up what we consider important modules. I'd love to tell sites "don't use Centos"

If a module is supported by the majors, I think it's save to just use it. But if it will most likely have to come from CPAN, then it needs a feature request and agreement by the project that the code is important enough to break installations on (most likely) Centos. I think that's a fair middle ground between "Use whatever the heck you want" and "Only use packages supported by all the distros". Not saying no, just justify it

-- GeorgeClark - 03 Nov 2016

This has gone way off the rails. I'll drop my "concern" if you could clarify the original proposal to include:

• A simple way to prevent all the magic for sites that are under constraints, for whatever reasons, to not use cpanm.
• (A way that should persist after installing a Foswiki upgrade. ie. Don't "surprise" the admin with silent installation of deps later.)
• Continue to follow the CodingStandards and FoswikiCodingConventions both of which say something to the effect of: If you find you do have to add a dependency from core code on an external perl module, make sure it is discussed to death first.

-- GeorgeClark - 04 Nov 2016

And everyone, please don't turn this into a discussion of coding standards that I referenced above. Yes, some of them are obsolete/incorrect. That's for another proposal or discussion.

-- GeorgeClark - 04 Nov 2016

OK, youre right. I edited and removed all my "off rails" comments.

-- JozefMojzis - 04 Nov 2016

Finally I've got all my work done and can clarify some of misunderstanding here.

First of all, let me explain the way I usually like things to be: easy for a user, flexible for a developer. When say 'user' it's not some selected kind one but any user. Hopefully it clearly explains that I won't limit anybody in ways to resolve issues like it would happen if installation blocked without Internet access. Yet, freedom is a two-side coin and I simply don't want the right of a corporation to jail their internal network to take my freedom to deliver a quality product to them and other users. It's just a matter of finding a well-balanced compromise.

Next question is the choice of modules to be used. Yes, I always was and will always be against the wheel reinvention (unless the other wheel doesn't fit the task or otherwise does not comply to my requirements). There is a point in choosing well-tested and reliable wheels. A single person couldn't consider all consequences of choosing a particular one without some help of the community. So, the content of the DEPENDENCIES file must be a subject to discuss among the developers – this is pretty much what George said above. But when the only obstacle for rejecting a module is it's absence in a damn-too-arrogant Linux distribution – that's unacceptable. Because eventually it may turn out to be more reasonable to spare some time for the personal life or a personal project instead of developing a new useful feature which would require to redo a code already existing somewhere else.

Now, down with common considerations and to the concerns raised. Current implementation of Foswiki::Aux::Dependencies will block nothing. It's just a helper. Period. The requirements I had for the module are:

1. It must run in any possible environment – with or without shell or Perl binaries, with or without any administrative access to the system. Actually, this part is the most important: the code is just a tool we can use whichever way we want. The discussion going on here is absolutely minor with this respect because any behavior is possible.
2. It must be simple and depend on nothing but what is already pre-included with Foswiki. So, I had to include cpanm script into tools/ as the only external dependency. The module itself is not even Perl's bare-bones OO but pure plain procedural style. So, it can be included into pretty much any Foswiki codebase.
3. It has to make life easier.

What is implemented in current version:

1. Dependencies check is enforced only and only if there is no $FOSWIKI_HOME/perl5/.checksums file. Mostly it would mean that it would be done once upon first start of foswiki.psgi script (considering that currently the implementation is branched from the OO line) on a fresh install. If a user removes the directory of just the .checksums file then check will be repeated.
2. But this means that if perl5/.checksums is created manually before the first run there will be no check performed and Foswiki will depend upon user installed modules. Any alternative way to block the first-time check could be considered and implemented too. Including the one where the Foswiki::Aux::Dependencies would try connecting to cpan.org port 80 and stop the checks if not available.
3. If upgrades check is needed once in a while it could be enforced either using a command line script tools/cpan_dependencies.pl or with help of configure web interface (to be implemented).
4. If a dependency fails to install then nothing special is done about it. cpanm output is recorded and stored for later use, but this is done anyway.
5. foswiki.psgi may fail to start but only due to missing dependencies. And this is for no doubt is something out of our hands – with or without automated install.
   • 500 error? It seems to be inevitable but the error can be catched and meaningful error page sent back to the user. With the proposed module we at least would have a good guess of what exactly went wrong and what modules are missing.
   • Otherwise the only way to determine if required installations have failed is to check $Foswiki::Aux::Dependencies::OK variable. Moreover, it will be true even if there are failed optional modules – for obvious reason; or if there failures for required modules for extensions not defined as mandatory ones (like JsonRpcContrib which is essential for the configure action).

Another significat subject is security matters and CPAN upgrades.

1. This might not be that important for some intranets. Just because they're isolated from outside access and because the company isn't that big and can trust it's every employee. They may ignore security problems in favor of available functionality.
2. For those who are concerned I can have few proposals:
   • Actually, support for installing packages by module names could later be implemented similarly to the currently abandoned tools/dependency_installer.pl script.
   • Support for installing from locally stored tarballs could be implemented too.
   • As it was mentioned once, a company could run their own CPAN mirror with only modules they need and patched the way they want them to.
   • And as a last resort cpanm allows installation of a particular module version. It's been used in the implemented module by simply passing versions strings from DEPENDENCIES file to cpanm command line. In other words, if there was known security issue with a Some::Module before version 1.01 then definition like '>1.01' will only install newer versions. If a module is volatile in its code quality but cannot be avoided then assuming that version 1.51 is the most reliable one we define it as '=1.51' in the dependencies.

The last item could also be supported by an extra local file which may contain user-defined module versions to be used instead of the defaults in Foswiki's DEPENDENCIES.

Hopefully I missed nothing.

PS. BTW, if anybody wants to look at real CPAN hell of zillions of dependencies drawing a project nearly unusable – go fetch Padre. That's the opposite pole I wouldn't want to have as a milestone to reach...

-- VadimBelman - 04 Nov 2016

Okay ... withdrawing my concern. As long as there is some way to prevent the installation of modules outside of distro package manager. I'd like it to be just a little more direct than the perl5/.checksums but it is enough.

-- GeorgeClark - 04 Nov 2016

I think it'll be better to have a $FOSWIKI_NOCPAN as a marker to block CPAN related activities. If it's set to 'startup' then only startup checks would be omitted leaving a possibility for later manual runs if there be code to do it via interface. Or if set to 'any' then the module would block just any attempts to install/upgrade with cpanm.

-- VadimBelman - 05 Nov 2016

Please don't depend upon ENV variables. Any site using SuexecUserGroup support in apache - to run the CGI under a different user, has no access to the ENV. It's sanitized to near oblivion.

-- GeorgeClark - 05 Nov 2016

File system approach is ugly. But they can happily co-exists so one can choose what suits him better.

-- VadimBelman - 07 Nov 2016

I'd like to see this initiative going forward somehow. However I am not sure the current implementation is modular enough to help the project. It seems it is part of Vadmin's Foswiki rewrite branch which is far from being ready to go.

Instead I'd like to see an approach that helps us now and which is not part of a total rewrite of Foswiki. We need it asap, best backwards compatible down to the Foswiki 2.1.x and 2.2.x branches.

The main objection against this proposal - as far as I am able to recap it - is that some organizations may block a Foswiki from auto-installing dependencies as part of their security policies. We really have to acknowledge this objection and provide a sensible alternative approach. For those situations I'd recommend to ship a virtual image or snap that comes with all dependencies pre-installed. People will understand that the feature set provided that way may be limited. At least they are able to get any Foswiki engine up and running.

Which leads to the questions of how to target Foswiki in the market: (1) enterprise level or (2) low level wiki. The point is that we need to regulate expectations. That is: you can't expect an enterprise level set of features without a certain extra amount of efforts.

Sometimes corporate IT strategies are at odds with business requirements. You can't solve this here! What we can do is to provide two separate alternatives, yet both need to be as easy to handle as possible, without one harming the other in terms of installation effort. Right now we hinder getting to the enterprise level by striving to please sub-optimal environments.

Sub-optimal environments are: windows, rpm linux distros, no network.

System packages vs self-compiled ones: there is no extra security provided by distributions, get over it. Most ship outdated perl packages with known security flaws that you only can hope distro devs backported security fixes. Just have a look at recent perl security in the area of unicode and regular expressions. Or take Email::Address security flaws that are not going to be fixed upstream at all. We either need to use Email::Adress::XS or switch off sending emails from within Foswiki if not available ... which is the case in almost all linux distros. However, sending email notifications is a base level feature even for low level wikis. We won't be able to provide this feature only using system-perl.

The problem of outdated perl and cpan packages in linux distros is not limited to the Foswiki community. For us the only way out is to let Foswiki install perl dependencies on its own locally and ship pre-installed images or snaps covering the basics. We already provide virtual machine images. So I less and less see the point of not implementing this feature.

-- MichaelDaum - 24 Aug 2018

You've mixed in a lot of things here. Let's separate a bit.

First of all, though the implementation was based on one of my branches, it doesn't depend on any of the OO-related code. The module is standalone and can be used as-is. Or at least it was my plan. If it's not then it must not be a big deal to fix it. So, it could be taken out as-is and put anywhere (after changing Aux to Util though).

Then, another mistake of yours is the perception of the idea behind the module. It was targeted at beginners who doesn't want to spend too much time fixing missing dependencies. It was for those who wanna unpack and play instantly. Because I suspect many potential users of Foswiki ditched it in favor of another solution simply because they were not able to get it up and running in 5-15 minutes. BTW, this is where PSGI version is great because one doesn't even need to configure a web-server to get a first succesfull run!

And, finally, having some container format is definitely a great thing. Even though I didn't use any of them yet... But that's different story unrelated to the above paragraph.

-- VadimBelman - 25 Aug 2018

Commenting is disabled while not logged in