Feature Proposal: Define Other Macro Expansion Types

Motivation

WikiDrawPlugin

In developing WikiDrawPlugin, I noticed that the raw TML is shown in the Changes listing - and it occurs to me that we could make Summary Rendering nicer by allowing MACRO's to define additional types/modes.

So, sometimes we want a macro to expand differently if its context (in the dictionary sense) on the page is "special". Let's call this the situation of a macro, to avoid confusion with IfStatements context.

In fact, a formal clarification might be:

1. In IfStatements context, we are talking about context as state information which pertains to a particular request. context is somewhat "global" and stable throughout processing of the request.
2. In this new feature, situation tries to capture pertinent information about every macro occurrence. Multiple occurrences of a given macro may see the same context information when they are processed, but different situation information which might influence their output or behaviour.

WysiwygPlugin

When we ask for TML2HTML, it might be beneficial (and allow richer interactive editing experience) if macros could be rendered (somehow [!] - WysiwygPlugin doesn't use Foswiki::Render ) with a special wysiwyg situation in mind.

BibliographyPlugin

A SEARCH summary result that uses the %CITE{foo}% macro - rather than assuming the requested topic's context - should instead assume the topic context that's passed in the situation stack (or similar mechanism)

Description and Documentation

IfStatements context is implemented as a flat hash of key:value pairs. Because situation is supposed to be sensitive to every occurrence of each macro; and the situation may change at various nesting levels that are possible with Foswiki macros; it is probably better to use a stack, with the idea that a macro's tag handler would be able to examine this stack.

The key point is to focus on the ability of one macro's output (return value) to carry with it a pointer into the situation stack. Then the parser can pass this on as input to the macros that are contained within the output. A %SEARCH% returning some text that contains occurrences of %WIKIDRAW% could set things up to make Foswiki process these %WIKIDRAW% macros with an additional argument to the registerTagHandler method or additional parameter to the params argument hash.

Additionally, it should be possible to push new situation onto the stack when we call the parser with $topicObject->expandMacros('text');

However, these two avenues have a severe limitation: the newly "pushed" situation applies to the entirety of the output of a macro or a piece of text to parsed. This may be unacceptable (Eg. the %WIKIDRAW% macro might occur in the header instead of format, where "Summary" situation is inappropriate).

So, instead of a tag handler returning text as the output, we should call something like

$parserObject->storeMacroOutput(
    'I am a piece of a search summary which mentions %WIKIDRAW% ...',
    [
        {
            name => 'Summary',
            emitter => 'SEARCH',
            situationHandlers => {
                pre => \&Foswiki::Macros::SEARCH::fooHandler, 
                post => \&Foswiki::Plugins::SEARCH::barHandler
            }
        }
    ]
);

for every piece of result text that needs its own situation

By default, most Foswiki macros would be processed with just one item pushed onto the situation stack: 'default'.

In Foswiki, we $topicObject->expandMacros('text')

Examples

For example, the %!WIKIDRAW{"drawing"}% macro would be able to define a 'Summary' view that renders as

[Wiki Drawing: drawing.svg]

additionally, macro's could optionally define a difference view, that allows better 'diff'-ing, and potentially, other rendering 'styles'

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

Unresolved points

• Callbacks, hooks. Can a macro that emits situation-specific output ask similarly aware macros to dispatch callbacks that it supplies? Namely pre/post per-macro expansion handlers, and perhaps pre/post per-'situation' expansion handlers
• Maybe we need to replace situation aware macros with a totally different registerTagHandler method - registerSituationalTagHandler perhaps? Or invent an OO base class, 'MacroClass-instance-per-macro' re-design... Whatever the case, what needs to be stored in the situation stack or passed in to the handlers:
  • situationTopic - Eg. so that BibliographyPlugin can know that it's in a 'Summary' situation on a particular topic that happens to be in the result set (vs the requested topic that initiated the search)
  • emittingMacroName - Plaintext name of macro that supplied the context
• Simple yet expressive data structures for the situation stack. So far the ideas aren't clear enough.
• Sorry Sven, for blowing out the complexity...

-- Contributors: SvenDowideit - 15 Sep 2010, PaulHarvey - 16 Sep 2010

Discussion

Sven, cool idea, I've been thinking about similar thing but specific to WYSIWYG. Might be cool to combine the two needs. I've made up some bogus spec, and added some creature feep

Feel free to knock it back to focus on your original concerns if this new stuff is bothersome.

-- PaulHarvey - 16 Sep 2010

I agree that this is a neat idea. Just a week or two ago I was wishing there was some way to force macros in EDITTABLE cells of type "label" actually render in table edit mode like they do in view mode - instead of not showing anything. I wonder if this feature could allow for something like that?

In case you're curious, the macros in question were used to display form fields from a topic named in another table cell - the cells containing the topic names were the only editable cells in the table.

-- LeilaPearson - 16 Sep 2010

Hi Leila, good use-case there. I think we're all dreaming of the same functionality.

Hmm, and now that I've tested the waters, I'm wondering if we should try for an OO tagHandler system (instance-per-taghandler).

Might make messing with callbacks cleaner, if the macro parser can be responsible for instantiating a TagHandler object for each particular %MACRO% (but not every occurrence). The TagHandler class would allow macro implementers to implement methods on their TagHandler objects, such as preExpansion, postExpansion, etc...

-- PaulHarvey - 16 Sep 2010

excellent - I like where you both are heading in the fleshing out - but I do hate situation

rendermode - is kinda better, as we don't actually need to squat on a short user-friendly word - this is a technical term describing a ARGH _not quite representational output type=.

additionally, taking Leila's point wrt editing - %!WIKIDRAW{"drawing"}% would have a 'view' rendermode that may contain an 'edit' button, an 'edit' rendermode, 'diff', 'summary' and 'wysiwyg' etc

however, this is really similar to a part of what context does - including that context isa kind of stack (um, from memory)

wrt instance per taghandler and callbacks - I think you'll need to spend a little time creating convincing arguments - I'm not sure the coding complexity is going to add sufficient value - we still need to make plugins simpler for not-quite-developers to throw together. (On the other hand, if TagHandlers are functors .... (yes, i know,same-same)).

ok - one bottom line that needs to be remembered: the Macro API needs to have a trivial way for people that are new to Perl and foswiki to create new Macros (ie, complexity needs to be optional additions that make sense to new hackers as they come to need them (unlike the Store and Search API's where it would be nice, but are deeper more complex by need)

(a little later)

urgh, your pre and post feels awful - because it'll become extremely difficult for non-experts to work out what happens when on a page containing lots of nested macros - it reminds me of what we had when everything was a commonTagsHandler - only even more complex.

-- SvenDowideit - 16 Sep 2010

Note that the original problem - rendering a macro differently depending on the context - can be handled by the context hash. You could always use entries in that hash as stacks, and push and pop state. We just don't have any standard definitions of those states e.g. "summary" state, or "normal render" state.

Maybe I'm being thick, but it really isn't clear to me what else is being proposed here. Are you proposing that the expansion of a macro should be passed information about the macro context it is evaluated in e.g. inside-out-left-right order suggests that the "inside-out" part knows what macro "context" a macro is being evaluated in? Or that there is some more global "context" that is like the context hash, but isn't? There seems to be another proposal, to wrap macro expansions (a la registerTagHandler) in some sort of context object. I'm not clear how this helps.

Altogether this to be a lot more specific before I can comment; right now it looks like a solution seeking a problem.

-- CrawfordCurrie - 16 Sep 2010

I'm not sure that we're at a point for the proposal to have concerns raised (yet); the clock isn't ticking. Just brainstorming.

Summary:

1. Is the Foswiki "context" hash really a stack, or a hash? Doesn't look like we have any push/pop methods.
2. When a macro results in text that contains other macros; it is this resultant text that needs to have additional context supplied to these resultant macros. The original macro cannot just use the existing context hash to supply context to the resultant macros:
   1. The context hash isn't targeted towards specific occurrences of macros.
   2. There is no way for the parser, macros, or any other mechanism that I know of for the context to "popped" once the parser is done expanding the resultant macros

So essentially, this situation thing - poorly named I agree, just a working name - in my mind, is set by the originating macro, and is passed on to the resultant macros by the parser.

As for why you would want to justify this - I am trying to solve problems that could benefit from context-sensitive macros; and it's proving difficult.

The simple example is a $summary() token, expanded from SEARCH format string, which supplies context to any macros that come out of the $summary() result (assuming they aren't escaped).

Somehow $summary() needs to tell the parser not to expand macros that don't implement 'Summary' context handler.

-- PaulHarvey - 16 Sep 2010

The context a hash, but the values in that hash can be anything, just so long as the users of that entry agree.

I understand what you are driving at, and you don't need anything clever to do this, just an agreement as to how a specific entry in the context hash is used. For example, you could have a context hash entry EXPANDING_MACRO that points to an array where you push the name of each macro name passed when looking for inner macros. Perl provides push and pop. But the proposal seemed to pack more than this punch, hence my confusion.

One further point about Schizophrenic macros. Macro expansion stacks can be quite deep, and it can be difficult working out why your macro doesn't expand the way you expected it to. Strong debugging tools would be essential.

Oh, and something else you need to think about. There are more ways for a macro to be "inside" another than meets the eye. Many macro implementations do "out of band" macro expansions on data they bring in (e.g. by calling expandCommonVariables). Other macros use "sections". It's not clear to me whether those expansions should inherit the situation of the calling macro.

-- CrawfordCurrie - 16 Sep 2010

"... whether those expansions should inherit the situation ..." is a very good question; and the reason why I thought the parser would have a common stack-thing across the whole parsing process. A nested macro might push a new situation, and macros nested within that could choose to walk the stack (or use an API function) to see if they're under 'WYSIWYG' situation even though the current stack top is 'Summary'.

Having said all this, I agree the potential for schizophrenia could be quite dramatic. Probably this approach isn't worth it.

I'll think about this for a while, maybe some others have ideas.

I'd like to hear from Sven how he envisaged a macro being able to target particular occurrences of macros with specific context.

-- PaulHarvey - 16 Sep 2010

mmm, I was 'simply' pondering the desire for some kind of modality for my tag - tbh, I still havn't thought about an implementation - I'm still wondering what could be done - and, as Crawford points out, if its a complexity too far.

more thinking from me too

-- SvenDowideit - 17 Sep 2010