not removed with $str =~ s/^\x{feff}//;
compare how the behavior changes with open my $fh, '<:encoding(UTF-8)', '../data/publicextract.charity.json' or die "Unable to read Charity JSON File"; compared to the open line you currently use.
If you want perl to treat the bytes in the file as UTF-8, and thus be able to use s/^\x{feff}/, you have to tell perl to read the file as UTF-8¹. If you want perl to continue to read the file as a series of bytes (not using the UTF-8 encoding), then leave your open as-is, and have your regex instead either search for the three bytes in octal with s/^\357\273\277// or in hex with s/^\xEF\xBB\xBF//.
#!perl
use 5.012; # strict, //
use warnings;
use Devel::Peek;
open my $fo, '>:raw', 'threebytes.bin';
print {$fo} "\xEF\xBB\xBF";
close $fo;
open my $fbytes, '<', 'threebytes.bin';
Dump($_ = <$fbytes>);
printf "length no-encoding: %d bytes\n", length($_);
printf "match no-encoding 3bytes? %s\n", m/^\xEF\xBB\xBF/ ? 'match' :
+ 'nope';
printf "match no-encoding unicode? %s\n", m/^\x{FEFF}/ ? 'match' : 'no
+pe';
close $fbytes;
open my $futf8, '<:encoding(UTF-8)', 'threebytes.bin';
Dump($_ = <$futf8>);
printf "length utf8: %d characters\n", length($_);
printf "match utf8 3bytes? %s\n", m/^\xEF\xBB\xBF/ ? 'match' : 'nope'
+;
printf "match utf8 unicode? %s\n", m/^\x{FEFF}/ ? 'match' : 'nope';
close $futf8;
__END__
SV = PV(0x6ac038) at 0xb3ebe0
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0xb4d308 "\357\273\277"\0
CUR = 3
LEN = 81
SV = PV(0x6ac038) at 0xb3ebe0
REFCNT = 1
FLAGS = (POK,pPOK,UTF8)
PV = 0x3442c98 "\357\273\277"\0 [UTF8 "\x{feff}"]
CUR = 3
LEN = 10
length no-encoding: 3 bytes
match no-encoding 3bytes? match
match no-encoding unicode? nope
length utf8: 1 characters
match utf8 3bytes? nope
match utf8 unicode? match
¹: or, not shown, use
Encode::decode('UTF-8', $octets) from
Encode
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.