Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

work with unicode data and regexps

by kamel (Initiate)
on Feb 10, 2003 at 09:06 UTC ( #234057=perlquestion: print w/replies, xml ) Need Help??

kamel has asked for the wisdom of the Perl Monks concerning the following question:

i have written a parser which searchs in a file for regexps. it all went fine but then i found out, that some of the data files are encoded in unicode. i tried several things from manuals and books but it just doesn't work. it seems that perl does not use the right encoding, it finds almost no regularexpression and the output is still with the NULs after every character. if i write the same lines in ASCII-File the keywords are found without any problem. i am using windows xp with active state perl 5.8. thanks vor your help. - testfile with the heart of the parser #open the file to search trough unless (open FILE, "<../perl/testproto/Protokoll.log") { die "Cannot open Keywordfile: $!\n"; } #search for the regexp in the file while (<FILE>) { chomp; if (/.*Cl(.*)/) { print $1; } else { print "Keine Treffer.\n"; } }
Part of the data-file: keywords to be found: Clinic, Surgon, Patient Code, ...
S: 07:02:36 ---------------------------------------------------------- +---------- S: 07:02:36 Clinic: Knie Klinik S: 07:02:36 Surgon: Prof X S: 07:02:36 Patient Code: Kiro S: 07:02:36 Size: 30 cm S: 07:02:36 Weight: 50 kg S: 07:02:36 Birthdate: Sa 1. Jan 2000 S: 07:02:36 Sex: male S: 07:02:36 Indication:

Replies are listed 'Best First'.
Re: work with unicode data and regexps
by allolex (Curate) on Feb 10, 2003 at 09:42 UTC
    You might want to try setting your locale to one that supports Unicode:
    use locale; use POSIX 'locale_h'; my $loc = "en_EN.utf-8"; setlocale(LC_CTYPE, $loc) or die "Invalid locale $loc";
    Or use a UNIX-GNU/Linux command like recode or iconv to convert your input to your character set. Hope that helps.

    Addendum: I'm not sure this will work on XP, but perldoc perllocale should be able to help you--and there is a Windows version of recode.

Re: work with unicode data and regexps
by dakkar (Hermit) on Feb 11, 2003 at 15:49 UTC
    open FILE,'<:UTF-16','../perl/testproto/Protokoll.log')


    binmode FILE,':UTF-16'

    If you know the encoding of the file, you can tell Perl what it is, and it will act accordingly. Otherwise, it assumes it is 1 byte = 1 character, and fail on several multi-byte (or variable-length) encodings.

    Note: I wrote UTF-16 above since you said you got 'NULs after every character'. You should check if it is indeed UTF-16, or maybe UCS-2 (they're not the same, you know...), or even something else.

            dakkar - Mobilis in mobile

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://234057]
Approved by jmcnamara
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2022-06-27 23:21 GMT
Find Nodes?
    Voting Booth?
    My most frequent journeys are powered by:

    Results (89 votes). Check out past polls.