You are here: Foswiki>Tasks Web>Item9845 (17 Dec 2011, GeorgeClark)Edit Attach

Item9845: Performance of language translations can be improved by using compressed .mo files instead of .po files

pencil
Priority: Enhancement
Current State: Closed
Released In: 1.1.4
Target Release: patch
Applies To: Engine
Component: I18N, Performance
Branches:
Reported By: OlivierRaginel
Waiting For:
Last Change By: GeorgeClark
As a user rightfully pointed out (please update this task to “Ἀπόδοτε οὖν τὰ Καίσαρος Καίσαρι καὶ τὰ τοῦ Θεοῦ τῷ Θεῷ”) Foswiki uses plain old .po files instead of generating .mo files.

I've quickly tried this on my system, and the performance is impressive according to NYTProf:
  • Without the patch:

Top 15 Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
172253.8ms108msLocale::Maketext::Lexicon::::importLocale::Maketext::Lexicon::import
161148.6ms48.6msLocale::Maketext::Lexicon::::CORE:readlineLocale::Maketext::Lexicon::CORE:readline (opcode)
11128.8ms34.9msutf8::::SWASHNEW utf8::SWASHNEW
802125.7ms376msFoswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 302ms)
4082117.8ms358msFoswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 144ms)
4466317.5ms22.8msFoswiki::Attrs::::new Foswiki::Attrs::new
11112.7ms112msmain::::BEGIN@22 main::BEGIN@22
124503412.7ms21.2msExporter::::import Exporter::import
111882112.3ms12.3msFoswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
22212.1ms40.9msFoswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
11112.0ms14.8msFoswiki::::BEGIN@576 Foswiki::BEGIN@576
801111.8ms12.8msExporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606241210.5ms19.3msFoswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5119.97ms11.8msCGI::::_compile CGI::_compile
23559.53ms11.2msFoswiki::::renderer Foswiki::renderer
From all 3329 subroutines

Source Code Files — ordered by exclusive time then name
StmtsExclusive
Time
ReportsSource File
1851109msline • block • subLocale/Maketext/Lexicon.pm (including 1 string eval)
2296284.0msline • block • sub/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
629848.4msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Render.pm
1037637.0msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
1015434.3msline • block • subutf8_heavy.pl
947632.0msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
859627.8msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
150020.4msline • block • subCGI.pm (including 2 string evals)

  • With the patch to use .mo files:

Top 15 Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
11128.8ms34.8msutf8::::SWASHNEW utf8::SWASHNEW
802125.3ms279msFoswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 295ms)
4082117.7ms260msFoswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 141ms)
4466317.4ms22.5msFoswiki::Attrs::::new Foswiki::Attrs::new
124503412.6ms21.1msExporter::::import Exporter::import
11112.6ms110msmain::::BEGIN@22 main::BEGIN@22
22212.0ms40.5msFoswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
111882112.0ms12.0msFoswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
11111.8ms14.6msFoswiki::::BEGIN@576 Foswiki::BEGIN@576
801111.8ms12.8msExporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606241210.4ms19.0msFoswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5119.87ms11.2msCGI::::_compile CGI::_compile
23559.45ms11.0msFoswiki::::renderer Foswiki::renderer
24118.82ms27.0msFoswiki::Prefs::Parser::::parseFoswiki::Prefs::Parser::parse
14537687.87ms7.87msFoswiki::::CORE:subst Foswiki::CORE:subst (opcode)
From all 3329 subroutines
Source Code Files — ordered by exclusive time then name
StmtsExclusive
Time
ReportsSource File
2296283.0msline • block • sub/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
629847.8msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Render.pm
1037636.3msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
1015434.2msline • block • subutf8_heavy.pl
947631.6msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
859627.5msline • block • sub/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
150019.6msline • block • subCGI.pm (including 2 string evals)

In order to make use of the patch functionnality, you have to run:
for i in locale/*.po;do msgfmt -o ${i%po}mo $i;done

-- OlivierRaginel - 19 Oct 2010


TODO:
  1. Integrate this into configure, as GeorgeClark suggested: A Checker for the po directory which regenerates the .mo file if newer dates found?

I'm going to draw a blank here - why would we want to do the compression in configure, vs. just compressing and shipping the files with the release, like we do with compressed css and js.

-- GeorgeClark - 10 Mar 2011

On IRC, Sven has pointed out many reasons to do this in configure. I have a proposed solution. Requires that we ship Locale::Msgfmt, otherwise we are platform dependent on msgfmt. Locale::Msgfmt is pretty small, pure perl.

If Internationalization is enabled, for each enabled language, if mo file doesn't exist, or is older than the corresponding po file, use msgfmt to compile the file.

#   new file:   ../../../CPAN/lib/Locale/Msgfmt.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/Utils.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/mo.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/po.pm
#   modified:   UserInterfaceInternationalisation.pm

-- GeorgeClark - 10 Mar 2011

The new compiler step has been checked into trunk only for now. If there is sufficient testing, we should check this into 1.1.3.

-- GeorgeClark - 12 Mar 2011

Also added an option to disable compression. (Needed for trunk.foswiki.org - or .mo files will become stale after svn updates.) Configure checker will warn if stale files are found and compression is disabled.

-- GeorgeClark - 12 Mar 2011

You mean it will warn if stale files are found and compression is enabled, right? Ah no, you mean if compressed files are found and compression is disabled. OK. But configure warns if files are stale (like .mo older than .po), and compression is enabled?

-- OlivierRaginel - 19 Apr 2011
 

ItemTemplate edit

Summary Performance of language translations can be improved by using compressed .mo files instead of .po files
ReportedBy OlivierRaginel
Codebase 1.1.3, trunk
SVN Range
AppliesTo Engine
Component I18N, Performance
Priority Enhancement
CurrentState Closed
WaitingFor
Checkins distro:01aa09a3f7f1 distro:aa0badb2cccd distro:beb29cfd8559 distro:1b266389e171 distro:98206ea3ecca distro:e7dd9585d074 distro:176ba318b883 distro:5e598dabeeff
TargetRelease patch
ReleasedIn 1.1.4
Topic revision: r18 - 17 Dec 2011, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy