Feature Proposal: Utilizing pure Perl for config specs.

Motivation

The ConfigurePlugin approach through extending Perl code with special comments semantics code is excessive and no flexible. Yet separation of actual LSC from specs deprives Foswiki core of some level of flexibility.

Description and Documentation

Put all basic processing of the config data into Foswiki::Config. This would include validation, smart processing or defaulting (wizards), etc.

The ConfigurePlugin would then have to be responsible for the UI only. In this job it would rely upon Foswiki::Config provided functionality.

In other words, lets do what classics say: separate UI from data and it's processing.

The target

This proposal is aimed at the new OO branch and further beyond it toward the new Extensions model where it would flourish at its best.

It is very-very preliminary.

Nothing is final until hardened in code. There is no code, no standards and only ideas are hovering around... I'm just trying to catch them and nail down to the whiteboard of this topic before they fly away.

So, don't take it too seriously as if it's gonna land in the core all of a sudden; as if in a month or few you get no chance but to accept. Think of a possibility and give your proposal on how to make things better.

The requirements.

• Be both UI and CLI compatible
• Support on-the-fly config changes
• Support the existing ConfigurePlugin as much as possible (specifically – wizards, checkers, etc.)
• Be transparent for the end-user, be ye olde cfg hash everybody is used to.

An implementation.

First, this is how it may eventually look in a complete Perl-only spec file (in terms of the new OO core):

    -section => Extensions => [
        -text => 'Description...', # Options are prefixed with '-' to distinguish them from keys. No key may start with '-'.
        -section => SomePlugin => [
            -text => 'For this section the order of keys is important.',
            '{Plugins}{SomePlugin}{ConfigFlag}' => {
                -type => 'BOOL',
                -label => 'To be or not to be?',
            },
            # Dot notation is easier to read.
            'Root.SubSection.PathToSomewhere' => {
                -type => 'PATH',
                -label => 'Where something is located',
                -wizard => 'Path', # Default prefix is Foswiki::Config::Wizards::
                -checker => 'Foswiki::Config::Checkers::PATH',
            },
            'Root.SubSection.Another\.Option' => { # Would we really want the dot in a key.
            },
            # Deep subkeying isn't the easy to read solution but might be used for complicated structures.
            Extensions => {
                SomeExt => [
                    TextKey => {
                        -type => 'TEXT(80)',
                        -label => 'Fixed text',
                        -default => '.Change this!.',
                    },
                    DynamicValidationKey => {
                        -type => '&Foswiki::Extensions::SomeExt::validateDynamic',
                        -label => 'This one is checked against a function',
                        -default => 3.1415926,
                    },
                ],
                OtherExt => {
                    Enable => {
                        -type => 'BOOL',
                        -default => FALSE,
                    },
                },
            },
        ],
        -section => 'Other Extension' => {
            -text => 'For this section the order of keys is irrelevant',
            -modprefix => 'Foswiki::Extensions::Other', # Will be appended with Wizards, Checkers, etc.
                                                        # Foswiki::Config or whatever is chosen to be system default would applied too.
            'Extensions.Other' => {
                Key1 => {
                    -checker => 'Checker1',
                    -depends => 'Key2', # Could be array ref is depends on more than one key.
                },
                Key2 => {
                    -default => '*not set*',
                },
            },
        },
    ],

The block is wrapped with $app->cfg->spec(...) call. Actually a developer may require some additional code to be run while the spec is read. In this case one would have to make the call manually.

This way specs doesn't get far away from the existing format but make the parsing and complicated structure of Foswiki::Configure::Item objects obsolete. Yet, as we deal with pure Perl here, it provides a new level of flexibility hardly possible with the current specs model without significant code changes and additional parsing.

Another advantage would be the fact that within the new OO model and with the new extensions there're ways to get a variety of backends for the config data. By imposing some limitations on option values (for example, no complicated data: only scalars, lists, hashes or easily serializeable objects) we could guarantee that even the specs themself could be stored as say JSON, YAML, whatever. Just have a corresponding backend written for the purpose. Would it be requested? May be yes, may be not. This is considerable.

But there is one application I have for the backends right now. There is a need for some configuration for the early stages of application life. This was previously done with setlib.cfg and LocaLib.cfg. But with the new config model it would be possible for a user to have all these settings being set with the configure UI (or CLI) absolutely transparently. From the developer point of view the only thing to be done is to declare specific backend for a key:

LibPath => {
    -backend => 'earlyPerl',
},
Debug => {
    -type => 'BOOL',
    -default => FALSE,
    -backend => 'earlyINI',
},

Early stage backends would be part of the core.

User side

There is a question I do expect: "What's the price of it all? What compatibility problems to expect?" The answer is: there must be no price; there must be no more compatibility problems then already introduced by the OO conversion. One can still use the notation:

$app->cfg->data->{Extensions}{Other}{Enable} = 1;

or even the old, to be avoided, global hash:

$Foswiki::cfg{Extensions}{Other}{Enable} = 1;

All the magic is hidden behind use of tied hashes as one could expect. How exactly this magic is done is not known yet. All the science is know about it is that this kind of magic is possible. But what is already know is how would it look to an end user:

my $wizard = $app->cfg->wizard( "Root.SubSection.PathToSomewhere" );

my $label = tied( $app->cfg->data->{Plugins}{SomePlugin}{ConfigFlag} )->label;

my @subKeys = $app->cfg->keysOf( 'Plugins.SomePlugin' );

my $value = $app->cfg->get( 'Path.To.TheOption' );

$app->cfg->set( 'Path.To.TheOption', "No" . $value );

And so on.

In particular cases it is possible to run with just plain LSC hash when it is read-only mode. But calling any of the corresponding methods on the cfg object would cause loadSpecs() method to be called and the data magic applied.

Note that calling the set() method would apply the magic; while simple:

$app->cfg->data->{Path}{To}{TheOption} = "No" . $value;

would not. But if later it would happen, and TheOption has a checker, and the value stored doesn't pass it (i.e. is invalid) then it would cause a late error report with no easy way to trace it down to the real location of the problem. This is the penalty for performance of the plain hashes. After all, LSC doesn't change during normal operation course. It's only the tests and configure to be worried about possible implications of the above code.

Yet, if tests are not intended to do a trick which requires the config data stay in plain form, they would be either demanded to call the loadSpecs() method or use set().

Error and other reporting.

Errors are to be reported by raising corresponding exceptions. But due to heavily used $reporter model within the ConfigurePlugin code it could be emulated to keep all existing wizards/checkers happy.

Same applies to NOTES.

The current specs format

Specs in the current format could be converted using a script. Or they could be read directly and mapped into the new format.

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: VadimBelman - 10 Nov 2016

Discussion

While me supporting your efforts as much as possible, this proposal makes me nervous a bit.

So, the the Foswiki should read the config as easily:

use YAML;
my $cfg = YAML::ReadFile('/path/to/some/file'); #returns a hashref
%Foswiki::cfg = %{$cfg}; #create the current HASH from the $cfg hashref

The above (with some small changes) actually works for me in my test-development wiki.

Please, make and use really _clean differentiation between 3 things:

• the serialized configuration file format (aka: how it is stored in the hdd) - now it is Data::Dumper format, would be nicer something more portable - by me: YAML.
• the in-memory format e.g.
  • like the current $Foswiki::cfg{some}{thing} hash
  • maybe some new as hashref , e.g. $conf->{some}{thing}
  • obj-interface $conf->get('some.thing')
• the configuration-management , which is now "the all other things around", e.g. wizards and like. The most important part of the config-management is the config.spec file.

The configuration management , e.g. using the informations from the "config.spec" (now regardless of the config.spec file-format) SHOULD NOT BE joined with the config-file itself.

So, trying to be more understandable:

1. we need an portable configuration-file-format. This file stores the actually configured values e.g. our current LSC.

• This should be easily readable with 3rd party tools too, therefore some commonly recognized format is better as the current Data::Dumper. By me: YAML (or json or xml - but both are harder-to-read as YAML and brings no other benefits.)
• This file MUST BE clean (as it is now) and MUST NOT contain anything what is currently in the config.spec.

2. we need the (a sort of) config.spec file.

• this MUST contain any(every)thing what need to know about the particular entry to allow verify the entered data-value. If you want, call it as "SCHEME".
• Because we trying to solve something what is usually called as: scheme based data-entry with user-defined data-types
• e.g. we defining a scheme for the data (aka the current config.spec)
• and this scheme is used by some software routines to limit and verifying the entered data (currently does this the configuration subsystem)

In your examples are some things like as label and so on. These are belong to the scheme (e.g. config.spec) and should NOT be mixed together with the config-values itself (e.g. LSC).

Also, your proposed "syntax". It is a "data-syntax" or the "scheme-syntax". E.g. the current LSC vs config.spec? What is its benefits above the "standard" perl-data-hash? Aren't you inventing again something "special" just because "we can"? Imho, it isn't more readable as any standard data-hash. Or it is meant as API? (aka subroutine calls?) then need the API definition.

So please, differentiate very clearly between the data-scheme and the data-value .

You probably want a good thing and maybe me doesn't understand it fully now. Therefore i wrote this comment.

-- JozefMojzis - 10 Nov 2016

You really should read:

• https://metacpan.org/pod/DefHash
• https://metacpan.org/pod/Sah
• https://metacpan.org/pod/JSON::Validator
• https://metacpan.org/pod/Kwalify
• and much more, like
• https://metacpan.org/pod/Data::Validate
• https://metacpan.org/pod/Data::Validate::XSD
• https://metacpan.org/pod/Config::Validate
• https://metacpan.org/pod/Type::Params

Even if we will not use the these CPAN-modules as whole, they're contains many great ideas /and steal-able / implementations about the "spec" (aka schema).

-- JozefMojzis - 10 Nov 2016

+1 on yaml

-1 on perl-only spec files

-- MichaelDaum - 10 Nov 2016

One thing is missed here. What is proposed is generally more about API for the specs and a way to handle both specs and the config internally. The proposal is for supporting the new Extensions.

A thing I didn't mention though had it in my mind: the new extensions allows to write your own config handling for whatever storage format one would like (my own preference is a DB for scalability). But current config implementation lacks a crucial part to make this scenario possible: saving. ConfigurePlugin despite being quite powerful tool has overcomplicated internals where actual data managements is tightly interlaced with UI. I wanna get the data out of it and send to where it really belongs – into Foswiki::Config.

The Perl code you see above displays the API for writing specs. BTW, I see no advantage in writing specs in YAML but that would be possible to implement too. Say, by starting the specs file with '#!yaml' special backend could be activated. It would translate YAML entries into the API hash format.

Jozef, either I wasn't clear enough or you misunderstood the concept. I thought that emphasizing the fact that access to the config would still be done in common hash format clearly separates specs from config data. The fact that config can be read as a plain hash does it too. Specs would exist separately as they do now. Yet, I don't want them to change too much because I wanna use the current configure infrastructure with minimal changes. Otherwise it's a job I cannot afford.

Specs and config data would intermix only in memory and only when requested directly or indirectly – i.e. when write operations are requested. But intermixing means only one thing: the internal structure would store not only value field for the config data but few other fields with this value's attributes. Keeping specs data separately is possible, of course, but it complicates algorithms of data traversal and analysis thus slowing them down.

So, to have a bit of summary:

1. Specs and config data are stored separately and may have different formats.
2. This all makes sense for the new extensions only.
3. Unless somebody wants to put his efforts into new configure tool the current ConfigurePlugin must be supported as much as possible.
4. The previous item must not prevent us from further extending the concept into new areas.

PS. Sorry for being a bit inconsistent. Doing several tasks at once doesn't help being really clear in expressions.

-- VadimBelman - 10 Nov 2016

A bit of almost real life example with the new extensions model. As soon as Foswiki::Config gets save() method I would expect that it using saveRecord() to store a singe config entry. The latter could be a pluggable one (as save() itself too but that's different story):

package Foswiki::Config;
...
pluggable saveRecord => sub {
     ...
};

A YAML support plugin would simply override it:

plugAround 'Foswiki::Config::saveRecord' => sub {
   my $this = shift;
   my %params = @_;

    yamlStoreEntry($params->{key}, $params->{value}, ...);
}

That's all, folks. BTW, in the code above if Foswiki::Exception::Ext::Last isn't thrown then the original saveRecord would do it's job to and LSC would be mirrored in two different formats.

PS. Just some food for the brains.

-- VadimBelman - 10 Nov 2016

Commenting is disabled while not logged in