You are here: Foswiki>Tasks Web>Item13566 (26 Sep 2016, MichaelDaum)Edit Attach

Item13566: Speed-aware reading encoded files

pencil
Priority: Enhancement
Current State: Confirmed
Released In: n/a
Target Release: n/a
Applies To: Engine
Component: Performance, Unicode
Branches:
Reported By: JozefMojzis
Waiting For:
Last Change By: MichaelDaum
Benchmarked different file-reading and decoding methods, and got some surprising results.

So asked an question in the Stackoverflow. http://stackoverflow.com/questions/31598938/benchmarking-utf8-file-read-explanation-of-the-differences

The answer pointed me to the module CPAN:Unicode::UTF8, it allows reading files 5 times faster as using Encode. The question contains some benchmarking code too.

Therefore, (and learned from the CGI-problem lesson too), IMHO Foswiki shouldn't use any CPAN modules directly, but thru its own interface modules, like Foswiki::Encode - instead of the plain perl's Encode:. Such approach could allow easily modify, replace the backend implementation, e.g. in this case the file-reading routine could be:

if( the_store_is_utf8 ) {
    read RAW file and use Unicode::UTF8 for the speed gain
} else {
    read the file using Encode
}

-- JozefMojzis - 27 Jul 2015

Setting this to confirmed. But does need more investigation.

-- GeorgeClark - 25 Sep 2016

Jozef, which perl version have you been benchmarking. Note that anything befor 5.22 is not worth it. I just looked at Unicode::UTF8's activity on github. ... which logs most recent commit back in 2013. Hm.

-- MichaelDaum - 26 Sep 2016
 

ItemTemplate edit

Summary Speed-aware reading encoded files
ReportedBy JozefMojzis
Codebase trunk
SVN Range
AppliesTo Engine
Component Performance, Unicode
Priority Enhancement
CurrentState Confirmed
WaitingFor
Checkins
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches
trunkCheckins
masterCheckins
ItemBranchCheckins
Release02x01Checkins
Release02x00Checkins
Release01x01Checkins
Topic revision: r3 - 26 Sep 2016, MichaelDaum
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy