Item9015: Using ATTACHMENTLIST in a query search causes the search to return only SOME of the results

pencil
Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a
Applies To: Extension
Component: AttachmentListPlugin
Branches:
Reported By: BigBaaadBob
Waiting For:
Last Change By: GeorgeClark
I have a web with about 3000 topics each of which has a 10 field form attached. If I use the following query:

%SEARCH{
type="query"
search="DocumentType='Document' AND DocumentStatus='Transferred'"
format="$percntATTACHMENTLIST{topic=$quot$topic$quot web=$quot$web$quot header=$quot   * [[$web.$topic][$topic]] has $fileCount files(s)$quot format=$quot      * $fileName$quot }$percnt"
nosearch="on"
}%

The search says:

Number of topics: 1267

But only about 80 show up in output. A whole bunch are missing.

Apache throws a slew of errors like this:

[Wed May 12 19:23:18 2010] [error] [client 192.168.254.192] [Wed May 12 19:23:18 2010] view: readdir() attempted on invalid dirhan\
dle $DIR at /var/lib/foswiki/lib/Foswiki/Store/RcsFile.pm line 256., referer: (redacted)/foswiki/bin/edit/(redacted)?t=1273717053;nowysiwyg=1                                                                    
[Wed May 12 19:23:18 2010] [error] [client 192.168.254.192] [Wed May 12 19:23:18 2010] view: closedir() attempted on invalid dirha\
ndle $DIR at /var/lib/foswiki/lib/Foswiki/Store/RcsFile.pm line 265., referer: (redacted)/foswiki/bin/edit/(redacted)?t=1273717053;nowysiwyg=1

If I change the search to specifically look for one of the missing topics, like this:

search="name='thetopicname'"

The search says "Number of topics: 1", but the topic isn't listed!

If I edit and save the topic and repeat this (single-topic) query, the topic does get listed.

I'm using AttachmentListPlugin (1.3.3, $Rev: 4282 (2009-06-22) $)

-- BigBaaadBob - 13 May 2010

I also get this error on the same lines in RcsFile.pm, but using DBCachePlugin. Problem seems to be DBQUERY trying to build a cache for a web that doesn't exist; but the problem is really hard for me to reproduce.

I discovered it was trying to open a non-existent web called "WikiTopics" or something similar, and assume my custom DBCacheContrib code (or usage of DBQUERY macro) is doing something bad.

I discovered the name of this phantom web using code similar to the following

--- /usr/local/src/Foswiki-1.0.9/lib/Foswiki/Store/RcsFile.pm   2010-01-18 01:16:20.000000000 +1100
+++ RcsFile.pm   2010-05-05 13:32:03.000000000 +1000
@@ -250,9 +250,9 @@
 sub getTopicNames {
     my $this = shift;
 
-    opendir my $DIR, $Foswiki::cfg{DataDir} . '/' . $this->{web};
-
     my @topicList;
+    if (opendir my $DIR, $Foswiki::cfg{DataDir} . '/' . $this->{web}) {
+    print STDERR 'Got dirhandle for web: ' . $this->{web};
     foreach my $f (readdir($DIR)) {
         # Validate and untaint
         next unless $f =~ /^(.*)\.txt$/;
@@ -264,6 +264,11 @@
     }
     closedir($DIR);
     return sort @topicList;
+    } else {
+      print STDERR 'Didn\'t get valid dirhandle for web: ' . $this->{web};
+    }
+
+    return;
 }

It's interesting to note that BigBaaadBob and I are both running Perl 5.10; or maybe that's just coincidence. I have noticed instabilities in DBCacheContrib since upgrading our server; spurious uninitialised variable warnings, and so on....

-- PaulHarvey

It turns out that the Apache errors were caused by the DBCacheContrib. I removed it, its plugin, and FormQueryPlugin from the system and the errors went away. See: http://foswiki.org/Tasks/Item9016.

Unfortunately, the above behaviour (missing topic) still remains. Further debugging shows that this happens when topics have auto-attached files. IOW, if a topic has auto-attached files, the query above will not return them in the listing even though the query knows about them.

The easiest way to reproduce is this:

  • Create a topic with an attached file.
  • Modify and use the above query to verify that it works on that topic.
  • Be sure you have Auto-attach enabled.
  • Edit the topic text on the server in an editor and delete the attachment meta-text.
  • Verify the attachment still shows up in Fiswiki (auto-attach is working).
  • Verify the above query does not list the topic.

I upped this to Urgent because ATTACHMENTLIST completely fails to work with auto-attachments.

-- BigBaaadBob - 13 May 2010

the ATTACHMENTLIST macro reads the list of attachments from the topic Metadata. The Func::getAttachmentList appears to internally iterate the attachment directory instead of just using Meta, but unfortunately in 1.0.9, the Func::getAttachmentList API does not yet exist.

Plugin needs to be changed to iterate the directory on disk and recover the attachment names from the actual files. eventually for 1.1, it could probably use the Func::getAttachmentlist to populate the list.

-- GeorgeClark - 13 May 2010

I've committed a fix and unit test to SVN,

-- GeorgeClark - 14 May 2010

Tested this and it seems to work fine, without any major performance issues.

-- BigBaaadBob - 18 May 2010

Great! Thanks for this improvement.

-- ArthurClemens - 18 May 2010
 

ItemTemplate edit

Summary Using ATTACHMENTLIST in a query search causes the search to return only SOME of the results
ReportedBy BigBaaadBob
Codebase 1.0.9, trunk
SVN Range
AppliesTo Extension
Component AttachmentListPlugin
Priority Urgent
CurrentState Closed
WaitingFor
Checkins AttachmentListPlugin:738b6fbdf4a7 AttachmentListPlugin:8ffb9ac03036 AttachmentListPlugin:672120ce52bc
TargetRelease n/a
ReleasedIn n/a
Topic revision: r12 - 26 Dec 2010, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy