Release Meeting: 18 May 2015

Details

  • Date: 18 May 2015 1300Z
  • Location: IRC #foswiki-release
  • Log: (See inline at end of this topic)
  • Participants (Add/Delete)

1. Character set issues: Tasks.Item13378

A lot of work has been done on this by CrawfordCurrie in the utf8 branch as a result of this fantastic work CrawfordCurrie was soaked by the team in virtual beer.

Jomo has done quite a lot of ad-hoc testing on these changes.

The consensus was to merge this back into master and move forward even though it's a relatively large change this late into a release it does fix some important issues. Some changes were requested to the utf8 branch to standardise using web= and topic= parameters for the enhanced SCRIPTURL and PUBURL macros.

GeorgeClark said that he will looking to create the next Beta with these changes merged later this week.

This is a major issue that is blocking Beta 2. From the task:
CGI returns all params (including path_info, url etc) in UTF-8 encoded byte strings. By happy coincidence, this is how perl internally represents UNICODE, so these strings appear to be UNICODE to most perl applications. Foswiki, in many places, does not decode these byte strings to the {Site}{CharSet} (topic and web name parameters, attachment filenames etc etc), However all strings output in the HTML must be output using the {Site}{CharSet}. So long as only 7-bit characters are used, this is not a problem, but as soon as wide-byte characters are involved, it all falls apart.

There are two approaches to solving this:
  1. Call Foswiki::UTF82SiteCharSet() everywhere a parameter value is destined to end up in HTML output
  2. Convert all param() etc inside Foswiki::Request so they are decoded to the {Site}{CharSet} before the ->param( call.

There will be cases where, through accident or design, the encoding currently "just works" or is explicitly converted. Some plugins (TopicInteractionPlugin for example) perform the conversion themselves, which makes a blanket conversion unworkable. Note that URI parameters are used in many places, and are even stored (e.g. in the cache, or in the CGI session) for future use.

Bottom line, what we currently have is badly broken for any {Site}{CharSet} other than utf8. We cannot release the way it is. I have started the process of (1) above, but it looks like there may be a lot of edge cases (e.g. usernames nd passwords; do we support wide byte chars in them?)

This is the one remaining major blocker.

2. Proposed Schedule for Alpha / Beta / GA

  • Beta 2: Merge in Item13378 changes from utf8 branch, later this week

Next actions

  1. When we merge, need to consider impact on trunk / release web sharing on t.f.o & f.o
  2. GeorgeClark to raise task ref possible sorting issue with EditRowPlugin and/or TablePlugin

Next meeting - - Monday Jun 01, 2015 1300Z — ReleaseMeeting01x02_20150601

IRC Log

14:39 Joining chat room...
Started talking in foswiki-release on Monday 18/05/2015 14:39:23
Started talking in foswiki-release on Monday 18/05/2015 14:39:23
14:40 -> jomo has joined foswiki-release
14:42 -> gac410 has joined foswiki-release
gac410 hi all ... i have a few minutes this morning, so figured I'd step in. Can't stay long though.
14:44 gac410 I won't be taking minutes or keep the logs, so someone else please handle that please.
14:46 CDot sure, can do, but there seems a lack of participation...
CDot has tried a couple of kicks on #foswiki, to no avail
JulianLevens1 No quite, it got me
14:47 gac410 looks like 9 in the chat room currently which is a pretty good turnout. Just everyone sleeping.
gac410 usually starts at 1300Z so it's a bit early.
gac410 but I can't stay long.
CDot y, probably wake up if we ping them MichaelDaum_ uebera Lynnwood jomo jast andreli
14:48 jomo im here..
CDot oh hell, we're on BST at the moment, I forgot.
MichaelDaum_ changes nickname to MichaelDaum
CDot hmmm google calendar says 1300 BST
14:50 gac410 CDot .. if you'd be the leader for today that would be great.
CDot MichaelDaum: you are the owner of the google calendar, are you not?
MichaelDaum sigh
MichaelDaum I emailed the edit access to everybody on the board once I created it
14:51 CDot and did you update the board since the last Association meeting?
14:52 andreli Hi, I was assuming 1300 GMT, that would be in about 8 minutes, no?
CDot you did. OK, I updated it.
CDot andreli: yes, but gac410 has limited time
14:53 JulianLevens1 Last meeting minutes: http://foswiki.org/Development/ReleaseMeeting01x02_20150504, no agenda for today?
jomo Next Release Meeting: Monday, 4 May, 1300Z ... bit late...
CDot JulianLevens1: would you mind taking the chair, please? I've been a bit out of the loop on release meetings, I'm afraid.
MichaelDaum guys: http://foswiki.org/Development/FoswikiCalendar
14:54 jomo bookmarked
MichaelDaum please bookmark shmockmark or do whatever you want with it
MichaelDaum but remember it and please help keeping it up to date
JulianLevens1 I'm not that much in the loop myself, but I'll do my best
JulianLevens1 First item would be http://foswiki.org/Tasks/Item13378
14:55 CDot only uses google calendar
CDot ah yes. Well, as you probably all know, we have a mostly unicode core, in the utf8 branch.
CDot There are still some known limitations; usernames cannot include wide-byte chars, neither can passwords
14:56 CDot apart from that, everything is unicode.
JulianLevens1 And jomo has tested it thoroughly?
CDot testing has consisted of: unit tests, user testing by jomo, and extensive code auditing by me.
jast how about compatibility with plugins?
14:57 CDot jast: core plugins are compatible, as are all plugins I personally maintain
CDot I cannot speak for any others
jast so third party extensions may need changes, right?
jomo tested it much - ofcourse nothing "systematic" - but found no bug (in the master i found many)...
CDot third party extensions may require changes, yes
14:58 CDot if, and only if, they do something stupid, though
jast s/stupid/necessary with old code/
CDot or they use an external 3rd party that isn't unicode friendly
CDot I added a flag - $Foswiki::UNICODE - that can be used to suppress encode/decode fairly trivially
14:59 CDot there's not much more than that I can do.
JulianLevens1 The $64,000 question is are you recommending this be incorporated into 1.20
CDot I am recommending it, yes.
JulianLevens1 gac410: Mr RM, and you say?
jomo me yes too - it has IMHO MUCH less bugs as current master
15:00 jomo talking as no RM
gac410 I support moving forward if others are comfortable. unicode / utf8 has been a huge lurking issue that's been deferred release to release since Twiki days
CDot well, there are a few other bug fixes, besides the unicode, but nothing most people would notice.
CDot mainly internal stuff.
15:01 gac410 I love seeing all the tasks marked needs merge.
JulianLevens1 This was listed as the last blocker, so is there anything else to discuss?
CDot plus I added %PUBURL and %SCRIPTURL extensions, so we don't have to compose external URLs as Attachment.gif
15:02 CDot http://trunk.foswiki.org/pub
CDot instead
jast fwiw I agree with merging. IMO it's better to do a few point releases (worst case) than never get this out
CDot or simply Atatchment.gif
jast same for *PATH and SCRIPTURL?
15:03 MichaelDaum http://trunk.foswiki.org/pub is that the specs we agreed on?
MichaelDaum I thought it was Attachment.gif
CDot MichaelDaum: probably not. It's what worked for me.
CDot doesn't recall a discussion on the topic
MichaelDaum could you please change it to topic= as thats what most other macros use
15:04 MichaelDaum path is inconsistent
jast yeah, we should make some effort to reuse terminology
CDot erm.... no. To refer to a web dir, you write http://trunk.foswiki.org/pub
CDot topic= in that case would be confusing
jast in that case the consistent, and ugly, way would be having both web and topic params
CDot I started with topic=, but changed to path= for that reason
MichaelDaum I can't remember we spec'ed for any http://trunk.foswiki.org/pub
15:05 MichaelDaum where would that point to
CDot you have to - otherwise you're back to writing http://trunk.foswiki.org/pub/Webname
MichaelDaum and why do I need it
CDot it's used in constructing partial URLs in the templates
CDot I didn;t analyse every use-case; I just saw it was heavily used.
jast shouldn't it have the full URL, then?
15:06 jast or in this case a full Web.Topic thingy?
MichaelDaum have you got an example url on github in foswiki/distro?
CDot erm, I'll check in the templates. mo.
jast I mean, http://trunk.foswiki.org/pub/Webname on its own isn't really valid anyway
MichaelDaum current best practice is web= and topic= for params of that kind
MichaelDaum as thats consistent
15:07 MichaelDaum so any combination of web= and topic= makes sense
MichaelDaum using normalizeWebTopicName of course
jast that's not compatible with this potential use case
15:08 jast normalize will turn web="Foo" and missing topic into "Foo.WebHome"
andreli CDot: Would http://trunk.foswiki.org/pub/Webname not be more consistent with other macros
jomo and what if someone write ... web="System" topic="Main.WebHome" ... - what happens? (sorry, only wondering)..
MichaelDaum jast, no topic= no need for normalization
CDot I don't have a problem with adding web= and topic=, so long as the rules for resolution are clear
jast jomo: usually the topic param overrides the web param
15:09 CDot the only use cases for the new code are in the unit tests, so easy to change
jast at least that's the standard normalize logic
jast so probably we'd be fine with web+topic and then normalize on that
JulianLevens1 CDot is it your argument is that it's a different use case, therefore using path= emphasises that distinction?
CDot I can't find the use case just now, but it's there; it was the result of a search
15:10 CDot I'm quite happy with web= topic=, if that's preferred
CDot it's not a big deal, IMHO
JulianLevens1 As long as the Docs make the different usage clear
15:12 JulianLevens1 So, are we in agreement to merge the utf-8 branch into master after the standardisation to web, topic parameters?
jast as always, the big discussions are about the small things
jomo from the wiki app dev perspecitve is better to have 1 arg and not make difference between web and topic - much many %IF mased on what i have (web or topic) - some cant use the "System" in the topic="..." - need use the web="..." - imho it is better to have only one "path" (at least have the path="...") as an alternative...
CDot y. I think the lack of unicode in user names is a much bigger deal
15:13 jast jomo: if both are used, you can just put your things in topic and ignore the existence of web
CDot jomo: the proposal would allos "topic=" to be used that way
jast assuming the developer done a decent job
jast which I am assuming in this case
jast anyway, yes, merge
CDot porn.gif
jomo
15:14 JulianLevens1 Any objections?
andreli no
MichaelDaum no
15:15 gac410 no
JulianLevens1 Ok, anything else to discuss
CDot ok, good, thank you.
jomo GRATZ cdot!
gac410 I should be back home late this week. I'll start a beta.
CDot please test master after the merge with as many evil things as you can think of
15:16 jomo will do ofc
CDot gac410: no hurry. going to have a shot at supporting unicode users too
gac410 When we merge, need to consider impact on trunk / release web sharing on t.f.o & f.o
CDot gac410: shouldn't have to, if we set $Foswiki::cfg{Store}{Encoding} it should "just work"
gac410 okay great
CDot t.f.o will croak on any failing conversion (not ideal, but there you go)
15:17 gac410 I have seen some extensions accessing the old Charset setting. so that needs a search, easiest in an old svn checkoug
CDot does anyone know where the constraints for the user validation wikiname are set up? (Javascript)
15:18 gac410 hm MichaelDaum ?
CDot y, I searched for the CharSet but didn't find any usages outside of core
CDot there will be some, though
gac410 I tripped over one. But don't remember where maybe it was a core ext.
CDot pretty much anything that uses AJAX will have done a decode
15:19 gac410 anyway, congratulations ... this is a fantastic move forward. Do we have a "virtual keg" option?
CDot
CDot it's just a shame it's not so "obvious" to end users. Limited advertising appeal.
15:21 jomo finally could start use the FW's full potentiall (as me is trying from 1.1.4)
gac410 Well it will be obvous to any user who has been dealing with scrambled characters. mis-sorted columns, etc.
15:22 gac410 Oh. I noticed on Jomo's uft8 test topic in TestCases, The one with tables, ssp, etc. The date colunn mis-sorts on the table. Really strange sorting.
CDot hadn't noticed that
gac410 I don't think that's utf8 isue though, maybe ERP or TablePlugin?
CDot maybe
CDot are you running utf8 branch?
15:23 gac410 lemme check ... have to start webserver it's down when I'm on wireless.
jomo here will be sorting issues (especially for the locale based sorting - e.g. in the sloval the "ch" (digraph comes after the "h" - but for this we will need to Unicode::Collate::Locale - and this sure isn't urgent... digraphs, trigraphs and tetragraphs are out of current FW scope...
15:25 gac410 Okay. TestCaseUtf8Errors on utf8 branch Date column. Sorts Dec, Apr, Feb, Dec
15:26 CDot jomo: with EditRowPlugin, sorting is done in JS, should already be locale-sensitive (albeit the locale set in the browser, rather than the server)
gac410 So it's not right, even if it was an alpha sort
CDot interesting.
CDot EditRowPlugin or TablePlugin sorting?
15:27 gac410 I think it's ERP, as there is no server interaction
15:28 CDot ok. please add a task, so we don't lose it
15:30 jomo arrrgh again "Warning: Updates found for 0 extension(s). tick Upgrade cross Ignore for 7 days"...
gac410 With that, I need to get going. Wake is this evening, funeral tomorrow. Thanks again CDot, jomo. Great work! I'll probably be back doing stuff late in the week.
15:31 JulianLevens1 Are we done?
CDot I'm done, thanks
<- gac410 has left foswiki-release
15:32 JulianLevens1 These other bits will be captured in the minutes and tasks raised
CDot thank you
JulianLevens1 I'll write it all up and attach the minutes
JulianLevens1 Any objections to closing the meeting?
15:34 JulianLevens1 Bangs gavel
jomo
JulianLevens1 Meeting closed, thanks everyone
Topic revision: r1 - 19 May 2015, JulianLevens
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy