Feature Proposal: Add list operators to FORMAT

Motivation

Macro FORMAT can do list rendering, but without list operations it's value is limited to display. SpreadSheetPlugin will fill this gap for some applications, but is not suitable for all situations.

For more dynamic applications it should be possible to process the lists. For instance you would want to process URL parameters from QUERYSTRING, or a list of topics.

Description and Documentation

Add list operations:

• sort items, reverse sort
• count items
• truncate list
• remove duplicate items
• remove items not meeting a condition
• get random item
• get item at index

How would the operator look like? Possibly similar to SSP: $SORT(list)

Examples

%FORMAT{"$SORT(beach plum,asparagus,citron)" }%

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: ArthurClemens - 16 Feb 2011

Discussion

Have a look at FORMATLIST from FilterPlugin. It already does all of the above operations except the last two (random item, indexed item) and a few more.

-- MichaelDaum - 16 Feb 2011

So do you propose to use the same syntax, and possibly code?

-- ArthurClemens - 16 Feb 2011

See AddFilterPlugin. We were nearly agreeing on that proposal to import FORMATLIST while leaving out some of the other features of FilterPlugin.

-- MichaelDaum - 16 Feb 2011

Yes, I do want to begin to add SSP functions to the Foswiki::Search::FormatResults code - that way (whem my cunning plan is completed) all macros that use a format= param will have access to that functionality automatically.

I guess I should see what code i can yoink from FilterPlugin, but as usual, the real work is in writing sufficient unit tests.

I'm going to add us as devs, cos we do all agree we need this functionality - what we need to decide is syntax (ie, first start is pasting the FORMATLIST format params, and then we can do the round of arguing, only to agree on what was first proposed

-- SvenDowideit - 17 Feb 2011

Another take is to look at the list operators in QUERY. While this may be too difficult syntax for the average macro user, I think the underlying capabilities should be aligned.

-- ArthurClemens - 17 Feb 2011

There's no formal notion of list objects in the core. The closest concept is that of result sets part of SEARCH. In that sense lists as a special kind of "result set" however not stemming from a SEARCH. We also have iterators of various kinds on perl level. It would make sense to leverage them to wiki apps in some way.

Here's what I'd like to do:

• parse in a list using PARSELIST{"some string" id="mylist" split="..." pattern="..." tokenize="..."} as defined in FORMATLIST, temporarily storing it in the variable mylist

then:

• format the list using FORMATLIST{["some string" | id="mylist"] header="..." footer="..." format="..." separator="..." sort="..." exclude="..." include="..." reverse="on/off" skip="..." limit="..." unique="on/off" selection="..." marker="..." map="..."} as defined in FORMATLIST as well

FORMATLIST either takes "some string" parsing it or processes a list already available under the given id parameter.

-- MichaelDaum - 17 Feb 2011

QUERY does have a concept of lists/arrays:

• Attachments: %QUERY{"length('System.FamFamFamContrib'/length(attachments[NOT lc(attr)=~'h'])"}%
• Topic revisions: %QUERY{"versions[info.author='SamuelJohnson'].info.version"}%

That would be useful if you want to access attributes of list elements. Not for lists of strings, but for instance for lists of topics (or lists of attachments).

Let's forget about those now.

I would be happy with the commands you have in FORMATLIST, except:

• I find map slightly confusing as I would expect it to use a function to operate on list elements
• selection should that not be match
• null - for other macros we use default
• sort should support multi key, meaning that a list type should be passed, see SupportMultiKeySorting

Some proposed commands map to FORMATLIST params and tokens:

• count: $count token
• truncate: limit and skip
• remove items not meeting a condition: exclude

Note sure:

• get item at index

Missing:

• get random item - seems silly but is really useful for displaying content items on the intranet homepage

-- ArthurClemens - 17 Feb 2011

mmm, ok, lets begin with FORMATLIST - the core already has that - its called FORMAT.

as for PARSELIST - really thats the same thing as making a typed resultset - where the type=string - so that is an interesting point. Crawford added a type=string to FORMAT for exactly that purpose - though only in the simplistic case atm.

What you're pointing out makes sense - the PARSELIST name is meh, but...

so really we go back to making FORMAT more complete, and adding more resultset types. wonderful.

-- SvenDowideit - 17 Feb 2011

Arthur, selection and marker are two quite useful concepts from WEBLIST. The map feature is something different. Getting an element at specific position X is done by a skip with limit=1. I can see that a randomizer for a list is quite nice actually, yes.

The list concept in QUERY is all good and fine, however the missing bit is that it needs a way to temporarily create a new list object that can then be accessed. Parsing a string to make it a list is the most obvious use case I can think of. Calling it PARSELIST seems to be straight ahead. Any other name for parsing lists?

FORMAT with a type=string is too dwarfish to be useful atm. Not sure if the outlined feature all fits into FORMAT. Separating out a list parser makes more sense IMO.

QUERY does fine on already well defined lists of property bundles like our attachments. This is less of a list but more of a keyed hash object mainly accessed by the name attribute, not by a sequence number.

-- MichaelDaum - 18 Feb 2011