Item13432: RCSLite fails with Malformed UTF-8 character

pencil
Priority: Urgent
Current State: Closed
Released In: 1.2.0
Target Release: minor
Applies To: Engine
Component: FoswikiStore
Branches: master
Reported By: GeorgeClark
Waiting For:
Last Change By: CrawfordCurrie
Unable to "manage" an existing attachment with RcsLite. RcsWrap works fine.

Malformed UTF-8 character (unexpected non-continuation byte 0x40, immediately after start byte 0xc3) in string eq at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 171.
 at /usr/lib/perl5/vendor_perl/5.20.1/CGI/Carp.pm line 379.
   CGI::Carp::realdie("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /usr/lib/perl5/vendor_perl/5.20.1/CGI/Carp.pm line 468
   CGI::Carp::die("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /var/www/foswiki/distro/core/lib/AssertOn.pm line 15
   Assert::__ANON__("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 171
   Foswiki::Store::Rcs::RcsLiteHandler::_readTo(GLOB(0x9154588), "\@") called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 287
   Foswiki::Store::Rcs::RcsLiteHandler::_ensureRead(Foswiki::Store::Rcs::RcsLiteHandler=HASH(0x9111af0), 3, 0) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 633
   Foswiki::Store::Rcs::RcsLiteHandler::getInfo(Foswiki::Store::Rcs::RcsLiteHandler=HASH(0x9111af0), 3) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/Store.pm line 310
   Foswiki::Store::Rcs::Store::_getAttachmentVersionInfo(Foswiki::Store::RcsLite=HASH(0x8ae7230), Foswiki::Meta=HASH(0x90996f8), 3, "CADEMIA_3.0-RC2.zip") called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/Store.pm line 320
   Foswiki::Store::Rcs::Store::getVersionInfo(Foswiki::Store::RcsLite=HASH(0x8ae7230), Foswiki::Meta=HASH(0x90996f8), 3, "CADEMIA_3.0-RC2.zip") called at /var/www/foswiki/distro/core/lib/Foswiki/Meta.pm line 1562
   Foswiki::Meta::getRevisionInfo(Foswiki::Meta=HASH(0x90996f8), "CADEMIA_3.0-RC2.zip", 3) called at /var/www/foswiki/distro/core/lib/Foswiki/Attach.pm line 168
   Foswiki::Attach::formatVersions(Foswiki::Attach=HASH(0x90e7350), Foswiki::Meta=HASH(0x90996f8), "size", 15191747, "attr", "", "path", "CADEMIA_3.0-RC2.zip", "user", ...) called at /var/www/foswiki/distro/core/lib/Foswiki/UI/Attach.pm line 95
   Foswiki::UI::Attach::attach(Foswiki=HASH(0x88ad2b0)) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 374
   Foswiki::UI::__ANON__() called at /usr/lib/perl5/vendor_perl/5.20.1/Error.pm line 419
   eval {...} called at /usr/lib/perl5/vendor_perl/5.20.1/Error.pm line 411
   Error::subs::try(CODE(0x804dea8), HASH(0x88bf8d0)) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute(Foswiki::Request=HASH(0x88a5fd0), CODE(0x887cf90), "attach", 1) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x88a5fd0)) called at /var/www/foswiki/distro/core/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x84470d8)) called at attach line 29.

-- GeorgeClark - 24 May 2015

I can upload the zipfile of the attachment and associated rcs files. It's big. >60 meg. RcsLite on 1.1.9 can also manage the attachment, all be it very slow.

-- GeorgeClark - 24 May 2015

Found a smaller attachment with same issue. (It seems I can't "manage" most binary attachments).

  • Item13432.tgz: Topic/Attachment that cannot be processed with RcsLite.

 /var/www/foswiki/distro/core (master)$ tar -tzvf Item13432.tgz 
-rw-r--r-- gac/users       635 2015-05-24 19:23 data/Sandbox/TestUpload.txt
-rw-r--r-- gac/users      1747 2015-05-24 19:23 data/Sandbox/TestUpload.txt,v
drwxr-xr-x gac/users         0 2013-01-25 12:21 pub/Sandbox/TestUpload/
-r--r--r-- gac/users      4070 2013-01-25 12:21 pub/Sandbox/TestUpload/woo%hoo.wav,v
-r--r--r-- gac/users      4070 2013-01-25 12:20 pub/Sandbox/TestUpload/woohoo.wav,v
-rw-r--r-- gac/users      3814 2013-01-25 12:21 pub/Sandbox/TestUpload/woo%hoo.wav
-rw-r--r-- gac/users      3814 2013-01-25 12:20 pub/Sandbox/TestUpload/woohoo.wav

-- GeorgeClark - 24 May 2015

I have a possible fix. I think the issue is that RCS files should always be read as bytes, and never go through utf-8 encoding/decoding. Things appear to work fine with this applied. But I'm not going to check it in, as I don't understand the consequences.

{state} = 'nocommav';
         return;
     }
-    binmode( $fh, $Foswiki::UNICODE ? ':utf8' : undef );
+    binmode( $fh ); #, $Foswiki::UNICODE ? ':utf8' : undef );
 
     my $state   = 'admin.head';    # reset to start
     my $term    = ';';

-- GeorgeClark - 27 May 2015

Y, that is not a fix. Working on it.

-- CrawfordCurrie - 27 May 2015

Can of worms opened, cleaned out and closed again.

-- Main.CrawfordCurrie - 29 May 2015 - 06:51
 

I Attachment Action Size Date Who Comment
Item13432.tgztgz Item13432.tgz manage 4 K 24 May 2015 - 23:26 GeorgeClark Topic/Attachment that cannot be processed with RcsLite
Topic revision: r7 - 29 May 2015, CrawfordCurrie
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy