Item10369: jquery.blockUI.uncompressed.js utf-8 encoded characters break YUI compressor

There are utf-8 characters in jquery.blockUI.uncompressed.js

The file contains 3 hex bytes ahead of the opening /*! that are not visible in various web views or in vi, but do show up in nano and can be seen with a hexdump

$ head -1 ~gac/jquery.blockUI.uncompressed.js | hexdump -C
00000000  ef bb bf 2f 2a 21 0a                              |/*!.|
00000007

$ hexdump -C (file containing /*!) 
00000000  2f 2a 21 0a                                       |/*!.|
00000004

The ef bb bf is bogus and breaks the javascript compressor

Generated /var/www/foswiki/trunk/JQueryPlugin/pub/System/JQueryPlugin/plugins/blockui/jquery.blockUI.init.js from /var/www/foswiki/trunk/JQueryPlugin/pub/System/JQueryPlugin/plugins/blockui/jquery.blockUI.init.uncompressed.js

[ERROR] 1:1:illegal character

[ERROR] 1:1:syntax error

[ERROR] 1:2:illegal character

[ERROR] 1:3:illegal character

[ERROR] 1:3:syntax error

[ERROR] 1:0:Compilation produced 5 syntax errors.
org.mozilla.javascript.EvaluatorException: Compilation produced 5 syntax errors.
        at com.yahoo.platform.yui.compressor.YUICompressor$1.runtimeError(YUICompressor.java:135)
        at org.mozilla.javascript.Parser.parse(Parser.java:410)
        at org.mozilla.javascript.Parser.parse(Parser.java:355)
        at com.yahoo.platform.yui.compressor.JavaScriptCompressor.parse(JavaScriptCompressor.java:312)
        at com.yahoo.platform.yui.compressor.JavaScriptCompressor.<init>(JavaScriptCompressor.java:533)
        at com.yahoo.platform.yui.compressor.YUICompressor.main(YUICompressor.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.yahoo.platform.yui.compressor.Bootstrap.main(Bootstrap.java:20)
Generated /var/www/foswiki/trunk/JQueryPlugin/pub/System/JQueryPlugin/plugins/blockui/jquery.blockUI.js from /var/www/foswiki/trunk/JQueryPlugin/pub/System/JQueryPlugin/plugins/blockui/jquery.blockUI.uncompressed.js

Removing the 3 bad characters with nano fixes the issue. This appears to be upstream corruption. I've downloaded the latest version from http://jquery.malsup.com/block/#download and the issue is still there. I can patch our version, but with the upstream corruption it will return on the next update.

Making this urgent because in some cases it blocks building of a release.

-- GeorgeClark - 17 Feb 2011

Babar pointed out that this is UTF8 encoding on that first line. yui compressor states:

  --charset character-set
      If a supported character set is specified, the YUI Compressor will use it
      to read the input file. Otherwise, it will assume that the platform's
      default character set is being used. The output file is encoded using
      the same character set.  IMPORTANT: if you do not supply this argument
      and the file encoding is not compatible with the system's default
      encoding, the compressor will throw an error.  In particular, if your
      file is encoded in utf-8, you should include this parameter.

I've tried changing my platform default charset encoding to en_US.UTF8, but the compress still fails.

-- GeorgeClark - 17 Feb 2011

The following patch to BuildContrib fixes the build on my system:

diff --git a/BuildContrib/lib/Foswiki/Contrib/Build.pm b/BuildContrib/lib/Foswiki/Contrib/Build.pm
index 2fb8ea7..bcf741d 100644
--- a/BuildContrib/lib/Foswiki/Contrib/Build.pm
+++ b/BuildContrib/lib/Foswiki/Contrib/Build.pm
@@ -1174,9 +1174,9 @@ sub _yuiMinify {
     my $cmd;
     
     if ($cmdtype == 2) {
-        $cmd = "java -jar $basedir/tools/yuicompressor.jar --type $type $from";
+        $cmd = "java -jar $basedir/tools/yuicompressor.jar --charset utf-8 --type $type $from";
     } else {
-        $cmd = "yui-compressor --type $type $from";
+        $cmd = "yui-compressor --charset utf-8 --type $type $from";
     }
     unless ( $this->{-n} ) {
         $cmd .= " -o $to";

-- GeorgeClark - 17 Feb 2011

These hex values are BOMs, a codepoint in unicode to mark endianness. We've got a few files that are in UTF-8+BOM, all of which are i18n files. It seems save to stripp off the BOM marker from blockUI as there's no need for the file to be encoded in utf8. The i18n files arent minified, that's why the yuicompressor did not bail out.

Reading the docu for charset it says

--charset character-set
      If a supported character set is specified, the YUI Compressor will use it
      to read the input file. Otherwise, it will assume that the platform's
      default character set is being used. The output file is encoded using
      the same character set.

So what if the file to be compressed is not in utf8 but still useds codepoints > 128 ? This might cause problems then. E.g. some German jquery developer called Rüdiger Fröhlich or so not saving his code in utf8. His copyright notion might look funny afterwards >:)

The reason I never have come accross this problem before is that my system's default encoding is utf8, whereas yours, George, might be not? In any case the compressed files should be in utf8 so setting the charset option to yuicompressor might be the only chance to get near a fix. It would be better if yuicompress would have separated reading some file in whatever encoding it is from writing the result in another encoding. Basically we have to know in advance which encoding is used by a js file before compressing it.

So we have to make sure that they are safe to be snarfed in using a global utf8

-- MichaelDaum - 18 Feb 2011

Things seem to be working okay for me.

-- GeorgeClark - 27 Feb 2011

Yea, I think we've got it. Let'S keep in mind to double check the charset of third party js files that we'd like to package for foswiki, not only in JQueryPlugin.

-- MichaelDaum - 27 Feb 2011