Nested search using ForEach and use of percnt in place of percent

Before I elaborate on my problem i have observed that $percent does not work as escape sequence with foreach plugin but $percnt does.... is it intentional?

We have Web for our maintaining catalogs. Each document is attached to a single page and each page has a form which contains the Category (formfield - ProLitProduct) , Make (formfield - ProLitMake) of item for the catalog. This is in addition to other formfields describing the document etc.

ProLitProduct - Select - list of products is maintained in topic of same name

ProLitMake - Select - list of makes is maintained in topic of same name

Each product can be from multiple makes and each make can have multiple products

What I want to do is create a hierarchical view of Catalogs Product -> make (makes applicable for that product -> Catalogs for the product and make

I tried using nested search but it becomes too cumbersome. Hence i have tried to use Foreach plugin. Enclosed is what i have been able to come up with

%FOREACH{"category" in="%SEARCH{ "^\|[^\*\|]*\|" topic="ProLitProduct" type="regex" casesensitive="on" nosummary="on" nosearch="on" multiple="on" noheader="on" nototal="on" separator=", " format="$pattern(^\| *(.*?) *\|.*)"}%" }%  
$category 
%FOREACH{"make" in="$percntCALC{"$LISTUNIQUE($percntSEARCH{ "ProLitProduct= '$category' " type="query" nosummary="on" nosearch="on" multiple="on" noheader="on" nototal="on" separator=", " format="$formfield(ProLitMake)"}$percnt)"}$percnt" }%  
$make 
$percntSEARCH{ "(ProLitProduct= '$category') AND (ProLitMake='ABB') " type="query" nosummary="on" nosearch="on" multiple="on" noheader="on" nototal="on" separator=", " format="$topic"}$percnt 
%NEXT{"make"}% 
%NEXT{"category"}%

Some how the inner search doesnot seem to work. I have looked at both variables $category and $make .... both have correct values at the given location. I have feeling i am messing up something in escape sequence. Could anyone suggest a alternate way ....

---

The good news is I can't see that you are doing anything wrong... the bad news is that ForEachPlugin is not very reliable, and isn't maintained much any more.

FOREACH also doesn't participate in normal inside-out, left-to-right expansion order because it's not a modernised "registerTagHandler" macro.

See Faq39 for more information about replacing FOREACH with FORMAT.

I would suggest you review nesting with sectional includes to simplify complex nested expressions. Something like (untested):

%SEARCH{
  "^\|[^\*\|]*\|"
  type="regex"
  multiple="on"
  topic="ProLitProduct"
  casesensitive="on"
  nonoise="on"
  format="$percntINCLUDE{\"%TOPIC%\" section=\"docategory\" category=\"$pattern(^\| *(.*?) *\|.*)\"}$percnt"
}%

<verbatim class="foswikiHidden">%STARTSECTION{"docategory"}%
   * %category%
%FORMAT{
  "%CALC{"$LISTUNIQUE(%SEARCH{
    "ProLitProduct='%category%'"
    type="query"
    nonoise="on"
    format="$formfield(ProLitMake)"
    separator=", "
  }%"}%"
  type="string"
  format="$percntINCLUDE{\"%TOPIC%\" section=\"domake\" make=\"$item\"}$percnt"
}%%ENDSECTION{"docategory"}%

%STARTSECTION{"domake"}%
      * %make%
%SEARCH{
  "ProLitProduct='%category%' AND ProLitMake='%make%'"
  type="query"
  nonoise="on"
  format="         * $topic"
}%%ENDSECTION{"domake"}%
</verbatim>

-- PaulHarvey - 06 Sep 2011

Note: personally, I usually avoid using type="regex" SEARCHes - I'd normally have a dedicated topic per row in your ProLitProduct table, each using a ProductCategoryForm or similar. Then the initial SEARCH would look more like

%SEARCH{
  "form.name='ProductCategoryForm'"
  type="query"
  nonoise="on"
  ...
}%

Then this type of SEARCH can automatically drive the allowable values for the Category field in your ProductForm definition.

Tip: FlexFormPlugin is very useful for presenting topics for edit, and JQGridPlugin is nice for displaying a set of topics' formfields as a grid.

-- PaulHarvey - 06 Sep 2011

---

Thanks Paul for your help. Tried out your nested search solution. It goes on processing for a while and no result appears. Is it because it is too much data ? ProLitProduct has 46 values ProLit Make has 138 Values and about 1533 pages with above combinations

When I tried to do nested search before, came upon similar problem, hence i went in for the foreach solution.

One solution would be to have Index per product and then a search to include all indexes as you suggested. But it involves creating these topics and then maintaining them everytime a product is added. I guess that seems to be the only solution in this case.

Is a function/plugin possible where the Formdata for all topics can be store in a single location and can be accessed ?

-- HarshadDivekar - 06 Sep 2011

If it's trying to compute hundreds of searches for thousands of hits, yes, it can become too slow. DBCachePlugin's DBQUERY (roughly equivalent to SEARCH{type="query") can speed things up a little (see Faq40 for adjusting queries), we're using the experimental MongoDBPlugin, but that won't be ready for general use until Foswiki 1.2 is done.

Try to use paging (pager="on") to limit the number of searches generated on a given topic view.

Or try to use the pattern described in Faq52 to maintain a static version of the products & category lists, if they aren't changing often.

See Faq56 for one way of removing some of the tedium when it comes to maintaining sets of DataForm topics

-- PaulHarvey - 06 Sep 2011

---

Thanks for your help Paul. I will try the above out. Will keep you posted.

-- HarshadDivekar - 07 Sep 2011

---

Tried using one page product wise, with separate searches. Works on each page individually but again conks off if I try to include the entire tree in one page. I think the search is too long to be done in one go. Hence have split the search (as you suggested earlier); one index per product, and one Index of indexes (w/o including the individual indexes in same page). hence only one search will run at a time. Then it works fine.

I was wondering though what is the limit at which search breaks? The documentation mentions you can go upto 16 levels. Mine was only 3 levels, there must be some other limiting factor. I don't think my classification data was too large; so you all must be encountering this problem frequently, is there a workaround ?

Also can parameter be added to search which will timeout in given time and prompt the user that the search is too complex to be computed? It will help people like me who cannot conclude the reason the markup is not working.

-- HarshadDivekar - 07 Sep 2011