Item5437: UTF-8 fixes for Foswiki 2.0 (was Foswiki 1.1 but deferred, was Foswiki 1.0 but deferred, was T4.2 but deferred)

Priority: Urgent
Current State: Closed
Released In: 2.0.0
Target Release: major

Applies To: Engine
Component: I18N, Unicode
Branches: Release01x01 trunk

Reported By: TWiki:Main.PeterThoeny
Waiting For:
Last Change By: GeorgeClark

two years ago i made (successfully) twiki 4 installation with utf-8 (one source modification required). but last week i couldn't make another utf-8 installation from twiki 4.2.

unsuccessies:

1. changing configuration parameters is not enough for unicode twiki installation. in my case, for correct handling utf-8 data, received from user, three .pm files required changes: lib/TWiki/{Save,View,Edit}.pm

1. for correct view within TinyMCE editor i had to make another change for correct handling javascript escaped unicode (from module URI::Escape::JavaScript)

[snip] twiki and similar systems should be configured to utf-8 as default, because it allows any people write in any language. [snip]

• With the correct setting for locale (en_US.utf8) and character set (utf-8), TWiki 4.2.0 has no problems displaying and raw-editing UTF-8 topics. This is basically the same as with Dakar, except...
• When editing a page using the WysiwygPlugin (i.e. either in TinyMCE or with the raw-edit link from TinyMCE), any multi-byte UTF-8 characters get converted to questionmark-icons.
• If I force UTF-8 I/O by changing the top-line of the TWiki edit script to "#!/usr/bin/perl -wTCS", UTF-8 characters are correctly displayed in TinyMCE. However, on reloading, switching to raw edit from TinyMCE or saving, UTF-8 characters still get converted to questionmark-icons. Moreover, direct raw-edit (using ?nowysiwyg=on) displays special characters incorrectly (they seem to be double-encoded).
• Forcing UTF-8 mode by adding the CS switches to view script also seems to give double-encoding problems. Using Ivan's patches instead (and twiddling with the locale and charset settings a bit) gives more or less the same results on viewing and editing.
• So it seems that most code handles UTF-8 correctly (or at least leaves it untouched), except the TML<->HTML conversions in WysiwygPlugin. Furthermore, it seems that TML->HTML can be made to work by forcing UTF-8, but HTML->TML always seems to break UTF-8 in our case.
• I am somewhat mystified as to why forcing UTF-8 leads to double-encoding problems though. Our hosting provider runs a (Debian-based?) Linux system with Perl 5.8.8. I've checked that the topic-datafiles are really valid UTF-8, and the locale used is present in the "locale -a" listing.
• The urlEncode/urlDecode roundtrip (in TWiki.pm, but called from WysiwigPlugin) correctly escapes UTF-8 characters to %uXXXX, but does not restore them. (Ivan's patch for TWiki.pm already fixes this.)

• Consider a Foswiki installation with {CharSet} = 'cp-1251'.
• We make an AJAX request to WysiwygPlugin's TML2HTML
• The responding HTML is correctly encoded cp-1251, and the http headers match.
• We're not using viewfile rewrites. Access to /pub goes directly to apache.
• There's an image tag to a topic + attachment both of which contain high-bit characters from cp-1251 (imagine that this part of this topic is cp-1251 native chars):

  <img src="http://ogg.lan/foswiki/a/pub/Sandbox/TеЙeЦДstеF/TеЙeЦ.png"/>

• The browser must make an HTTP request for the image. But as per rfc3986, the high-bit characters must be percent-escaped first. According to the apache logs, the request URI looks like this:

  http://ogg.lan/foswiki/a/pub/Sandbox/T%D0%B5%D0%99e%D0%A6%D0%94st%D0%B5F/T%D0%B5%D0%99e%D0%A6.png

  - which is the previous string, in unicode/utf-8.
• This is because there's no way of encoding what charset the escaped octets come from, in the URI itself. So the convention is that the browser converts to unicode/utf-8 first, and escapes that.
• Apache saw a request for a filename that doesn't exist on-disk. On-disk, we only have the cp-1251 encoded filename, because that's what Foswiki saved it as. If apache somehow knew that the URI was utf-8/unicode (it cannot), and apache knew that the on-disk filenames are cp-1251, there might be a chance there could be some conversion.... however as-is, we suffer a file-not-found error.
• Instead, what works is escaping "early" (with the initial rendered HTML). This works:

  <img src="http://ogg.lan/foswiki/a/pub/Sandbox/T%e5%c9e%d6%c4st%e5F/T%e5%c9e%d6.png"/>

  - the browser didn't have to escape anything, because there were no high-bit chars; and apache didn't bother trying to understand what charset the URI was in, it just un-escaped the octets which happened to match byte-for-byte the cp-1251 filename on disk.

• Adding selenium-based tests. The Foswiki::Engine in use, and the webserver config, and the versions of CGI.pm etc., the store/listeners that are in use, all seem to be significant variables in a successful utf-8 Foswiki. And I keep breaking mine.
• So I'm making a test generator. Checkins soon.
• I hope to make it verify over all available Foswiki::Engines
• To simulate MongoDBPlugin and (I assume) DBIStoreContrib behaviour, I want to add a special experimental configure setting to RcsLite=/=RcsWrap which will return (decoded) "unicode" perl character data, rather than octets.
• Need to act on Item1344, add charset to TOPICINFO
• Needs a new configure setting, for which charset should be assumed when TOPICINFO.CharSet is missing. But it's not that simple:
  • What about externally generated Topic.txt which omit the TOPICINFO completely
  • What about topics which now have the TOPICINFO.CharSet, but older revisions don't
  • Can we say: "When TOPICINFO.CharSet is absent, BUT TOPICINFO.date exists, AND that TOPICINFO.date is older than (some $Foswiki::cfg{Site}{Default}{CharSet}{OlderThan}{$epoch} setting), then assume the encoding is the value of $Foswiki::cfg{Site}{Default}{CharSet}{OlderThan}{$epoch}
  • Can we also say: "Assume the topic to be in $Foswiki::cfg{Site}{CharSet} encoding otherwise"

• No need for a TOPICINFO.charset, because the charset is always UTF8.
• Histories are converted to UTF8.

---++ Module Failure summary
FormattingTests has 1 unexpected results (of 90):
   * F: FormattingTests::test_Item11671
SemiAutomaticTestCaseTests has 1 unexpected results (of 24):
   * F: SemiAutomaticTestCaseTests::test_TestCaseAutoInternalTags
TWikiFuncTests has 1 unexpected results (of 24):
   * F: TWikiFuncTests::test_getExternalResource
ViewFileScriptTests has 1 unexpected results (of 14):
   * F: ViewFileScriptTests::test_simple_textfile
ResponseTests has 2 unexpected results (of 9):
   * F: ResponseTests::test_body
   * F: ResponseTests::test_empty_new
ExpandMacrosTests has 2 unexpected results (of 11):
   * F: ExpandMacrosTests::test_delayedExpansionInline
   * F: ExpandMacrosTests::test_delayedExpansionInlineTypeString
NetTests has 2 unexpected results (of 5):
   * F: NetTests::verify_getExternalResource_Sockets_HTTPResponse
   * F: NetTests::verify_getExternalResource_Sockets_noHTTPResponse
RenderFormTests has 2 unexpected results (of 8):
   * F: RenderFormTests::test_render_for_edit
   * F: RenderFormTests::test_render_formfield_with_form
UIFnCompileTests has 2 unexpected results (of 78):
   * F: UIFnCompileTests::verify_switchboard_function_compare
   * F: UIFnCompileTests::verify_switchboard_function_compareauth
RenameTests has 4 unexpected results (of 29):
   * F: RenameTests::test_referringTopicsThisWeb
   * F: RenameTests::test_renameTopic_find_referring_topics_in_all_webs
   * F: RenameTests::test_renameTopic_new_web_same_topic_name
   * F: RenameTests::test_renameTopic_same_web_new_topic_name
ConfigureTests has 8 unexpected results (of 22):
   * F: ConfigureTests::test_Package_install
   * F: ConfigureTests::test_Package_makeBackup
   * F: ConfigureTests::test_Package_sub_install
   * F: ConfigureTests::test_UI
   * F: ConfigureTests::test_conflict
   * F: ConfigureTests::test_loadpluggables
   * F: ConfigureTests::test_parseSave
   * F: ConfigureTests::test_resection
TableParserTests has 8 unexpected results (of 8):
InitFormTests has 10 unexpected results (of 10):
WysiwygPluginTests has 10 unexpected results (of 18):
EngineTests has 10 unexpected results (of 12):
ExtendedTranslatorTests has 22 unexpected results (of 63):
HTMLValidationTests has 32 unexpected results (of 106):
TranslatorTests has 98 unexpected results (of 390):

2644 of 2868 test cases passed(2640)+failed(4) ok from 2881 total, 13 skipped
0 + 224 = 224 incorrect results from unexpected passes + failures
1..70907