Re: unicode version of readdir

Well I can certainly qualify as a un*x-oriented monk (but I am no zealot;) IMVHO here we are talking filesystem user interface (and only that -- meaning using ONE env var for TWO (or more purposes) does not feel right):

latin-1-like-aware filesystems have "forbidden chars" (NTFS quite a lot I think). What does happen with unicode-aware filesystems?

Perl should ideally allow a default that makes sense for the platform but IFF the default is sensible. It's better if the default is the same for for all OS. If there is no consensus, then some kind of pragma will be needed like 'use feature filesystem qw(unicode-aware ...)'

Feature can be promoted to default once unicode-aware filesystems are really stable y/or their semantics/implications clearly understood

There is always the option of supporting an OS-dependent version of the needed calls (for unicode-aware filesystem). Not ideal but often necessary in the case of conflicting abstractions.

cheers --stephan

Comment on Re: unicode version of readdir

Replies are listed 'Best First'.
Re^2: unicode version of readdir by dk (Chaplain) on Sep 14, 2007 at 12:41 UTC
Oh. I'm absolutely against of making unicode-aware behavior by default, be it even really really stable. Filesystem business is external to perl, as well as locales and IO. Actually, perl does really well with the latter two - one says explicitly "use locale" and one uses IO layers, and that's it. What I propose is some orthogonality in this regard, say "use feature filesystem => 'utf8'" and enjoy utf8 input from readdir, and possibly from open. update: 'use feature' and the like are global, possibly a three-arg 'opendir' or two-arg 'readdir' would be better? Or, like 'binmode FILEHANDLE', something like 'utf8mode DIRHANDLE' would be more interesting?	[reply]
Re^3: unicode version of readdir by demerphq (Chancellor) on Sep 17, 2007 at 17:46 UTC
Personally i disagree with this. Id like to see this behaviour be automatic. Both VMS and Win32 support filesystems that properly support unicode file names, unlike the hacky approach of the UNIX world. Since we can know at a system level how the data is encoded UTF16LE there is no problem mapping that data to and from UTF8. The issue you mentioned elsewhere of codepages doesnt play a role as far as i understand it (which could just mean I misunderstand it :-). Unicode data is unicode data. Codepages only play a role when you want to translate a unicode string into your current "locale", but if you stay in unicode the entire time it shouldnt matter. The problem here is that the internal interfaces are designed around an exceptionally simple interface due to legacy reasons. However were someone to put in the work to change all of these interfaces to deal with SV's and not char 's then we would be prepared for the existing OS/filesystems that can handle unicode filenames properly as well as for when some nix file system does the same thing. If you check the archives you will see this subject has come up before and that Jan Dubois (of Activestate) has opined on various pathways to resolve it, however they are large scope projects which are not within scope of 5.10, although could easily be in scope for 5.12. --- $world=~s/war/peace/g	[reply]
Re^4: unicode version of readdir by dk (Chaplain) on Sep 18, 2007 at 20:34 UTC
There's no problem when you send a FS request (open,stat etc)-- this can easily be automatic. There's a problem when you receive FS info, that is, OS cannot decide for you what format to provide data in. For example, on win32 you might not want to have results in unicode, -- so OS conveniently maps internal UTF16LE names into, say, Latin-2. Of course, if we (suppose) never use 8-bit OS interface, but only unicode API, we convert utf8<->bytes by using perl's Encode, but then another problem arises: other users of the OS may (externally) supply us with a 8-bit file name, and if system codepage is tricky, then there's no way we can correctly convert the supplied 8-bit filename into proper UTF8 using Encode. I actually just 20 minutes ago send a mail to p5p with a proposed patch that does exactly this.	[reply]


P is for Practical
	PerlMonks