Re: Re: Sorting utf-8

>current locale if use locale is in effect. See perllocale.

But locales and unicode don't mix well:

perldoc perlunicode:

"Use of locales with Unicode data may lead to odd results.
Currently,Perl attempts to attach 8 bit locale info to characters
in the range 0..255, but this technique is demonstrably incorrect for
locales that use characters above that range when mapped into
Unicode.  Perls Unicode support will also tend to run slower.  Use of
locales with Unicode is discouraged."
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I would have thought Unicode::Collate is the correct way to go when sorting utf8 encoded data. But, looking at the docs, I couldnt make head or tail of it - it assumes you to know an awful lot about "Unicode Technical Standard #10"

If your current locale is, say, es_ES, how do you actually instantiate the correct Unicode::Collate object for that locale?

Comment on Re: Re: Sorting utf-8

Replies are listed 'Best First'.
Re: Re: Re: Sorting utf-8 by dakkar (Hermit) on Apr 24, 2003 at 11:58 UTC
Looking at the docs (and guessing at the UTS#10), I'd say that the Unicode collation algorithm is locale independent: it is supposed to give a collation key for any Unicode string (the fact that they are encoded in utf-8 is immaterial, BTW) So I'd just use `Unicode::Collate->new()->sort(@list)` If you want to customize the results, then you'll have to understand the UTS#10, but otherwise it should "just work" -- dakkar - Mobilis in mobile Most of my code is tested... Perl is strongly typed, it just has very few types (Dan)	[reply] [d/l]
Re: Re: Re: Re: Sorting utf-8 by Anonymous Monk on Apr 24, 2003 at 12:54 UTC
Hmm, pretty cool if that's how it works - from looking at the constructor arguments I was wondering if you'd need a hash defining per-locale options, but maybe not. I would pay a LOT of money for a book that explained Unicode in a non-geeky manner :)	[reply]
Re: Re: Re: Sorting utf-8 by webelan (Acolyte) on Apr 24, 2003 at 13:37 UTC
Thank you, and also Dakkar, plus all the other kind people who posted, The solution is indeed to use "Unicode::Collate" together with a file called "allkeys.txt". No locales needed. Just put "use unicode::collate" and add the lines: `my %tailoring; my $Collator; $Collator = Unicode::Collate->new(%tailoring); @char = $Collator->sort(@char);` [download] and sorting works "automagically"; French is now totally correct; for the other character sets such as Greek it looks logical but I'll get our translators to check the order for me just in case there are still some quirks. However, Swedish and Finnish no longer sort correctly, because in those languages Ä, Ö etc are considered to come after "Z" so it looks like I'm going to have to do an if/else with "normal" sort and "collate". But who cares, I'm a huge step forward from where I was this morning. Thanks a lot guys, Anne	[reply] [d/l]
Re: Re: Re: Re: Sorting utf-8 by Anonymous Monk on Apr 24, 2003 at 15:08 UTC
Good to hear its kind of working for you! But there will be a correct way for handling Swedish and Finnish unicode collation, so I wouldn't start switching between sort and collate in such cases until you've exhausted the "correct" way(s?) of doing this. Perhaps you could ask SADAHIRO Tomoyuki, the Unicode::Collate author? It would certainly be good if you can post any reply you get here, since this is the kind of stuff perl developers are going to have to know more and more about - I'm of course talking about the folks who don't know it already :)	[reply]
Re: Re: Re: Re: Sorting utf-8 by richyboy (Acolyte) on Apr 24, 2003 at 21:58 UTC
Sorting is not going to work for all locales simply by using allkeys.txt sadly. To start understanding, try looking at http://oss.software.ibm.com/icu/userguide/Collate_Intro.html http://oss.software.ibm.com/icu/userguide/Collate_API.html has some useful links as well. You'll also have to look at the official Unicode consortium docs as well. http://www.unicode.org Not really much fun I'm afraid :)	[reply]


laziness, impatience, and hubris
	PerlMonks