Re^2: UTF-8 text files with Byte Order Mark

Yeah, this works, except that the BOM indeed is a three-bytes thing as said above. So the code, that seems to work, now looks like this:

while (my $line = <$rulesFH>) {

    if ($. == 1) {
        # Remove Byte Order Mark if it's there
        use Encode;
        my $octets = encode("utf8", $line);
        $octets =~ s/^\x{ef}\x{bb}\x{bf}//;
        $line = decode("utf8", $octets);
    }

    # rest...
}
[download]

Comment on Re^2: UTF-8 text files with Byte Order Mark Download Code

Replies are listed 'Best First'.
Re^3: UTF-8 text files with Byte Order Mark by ikegami (Patriarch) on Feb 13, 2007 at 20:48 UTC
`my $octets = encode("utf8", $line); $octets =~ s/^\x{ef}\x{bb}\x{bf}//; $line = decode("utf8", $octets);` [download] is the same thing as `my $BOM = decode("utf8", "\x{ef}\x{bb}\x{bf}"); $line =~ s/^$BOM//;` [download] is the same thing as `my $BOM = chr(0xFEFF); $line =~ s/^$BOM//;` [download] is the same thing as `$line =~ s/^\x{FEFF}//;` [download] which is what I gave you. Much simpler!	[reply] [d/l] [select]
Re^4: UTF-8 text files with Byte Order Mark by muba (Priest) on Feb 13, 2007 at 21:01 UTC
Meh. Indeed, I didn't realise that. Thank you!	[reply]
Re^4: UTF-8 text files with Byte Order Mark by Anonymous Monk on Sep 30, 2011 at 18:30 UTC
Thank you!!! This saved me a lot of trouble. I am also trying to strip out these UTF-8 byte order mark characters, which google docs puts in by default to downloaded text files. By the way I found that \x{FEFF} was not the same as \x{ef}\x{bb}\x{bf}	[reply]
Re^5: UTF-8 text files with Byte Order Mark by ikegami (Patriarch) on Oct 01, 2011 at 21:53 UTC
By the way I found that \x{FEFF} was not the same as \x{ef}\x{bb}\x{bf} Yeah, "\x{ef}\x{bb}\x{bf}" is the UTF-8 encoding of the BOM / U+FEFF / "\x{FEFF}".	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.


Just another Perl shrine
	PerlMonks