http://qs321.pair.com?node_id=870930


in reply to Re^2: Cleaning up non 7-bit Ascii Chars for XML-processing
in thread Cleaning up non 7-bit Ascii Chars for XML-processing

00-1F and 7F aren't printable, so I escape those too.

And other than those things, isn't that really equivalent to what I was doing?

Yes, it's the same except where it's not.

Except that, it appears you are outputting ’ where I was outputting \, and are those in fact equivalent?

So you still haven't fixed your bug. Start with that.

and are those in fact equivalent?

No. I guess that's yet another bug.

Replies are listed 'Best First'.
Re^4: Cleaning up non 7-bit Ascii Chars for XML-processing
by ikegami (Patriarch) on Nov 11, 2010 at 21:06 UTC
    ...but it seems that I misread. I thought you were generating the XML.

    The XML is always output as "UTF-8"

    No it isn't.

    "’" is "E2 80 99" in UTF-8.
    "’" is "92" in cp1252.

    You've indicated you have the latter.
    You've indicated the document claims to be the former (implicitly).

    You can either fix the encoding, or fix what the XML says the encoding is. The former is easier.

    use strict; use warnings; use Encode qw( encode decode ); sub fix_broken_text { my ($self, $field) = @_; $field =~ s/&/&amp;/g; $field =~ s/</&lt;/g; $field =~ s/>/&gt;/g; $field =~ s/"/&quot;/g; $field =~ s/'/&#39;/g; return $field; } my $decoded_xml; { open(my $fh, '<', $xml_qfn) or die; binmode($fh); local $/; $xml = decode('cp1252', scalar(<$fh>)); } ...Try to fix problems with unescaped characters... my $encoded_xml = encode('UTF-8', $decoded_xml); ...Pass $encoded_xml to parser...

    If only parts are cp1252,

    use strict; use warnings; use Encode qw( encode decode ); sub fix_broken_text { my ($self, $field) = @_; $field = decode('cp1252', $field); $field =~ s/&/&amp;/g; $field =~ s/</&lt;/g; $field =~ s/>/&gt;/g; $field =~ s/"/&quot;/g; $field =~ s/'/&#39;/g; $field = encode('UTF-8', $field); return $field; } my $encoded_xml; { open(my $fh, '<', $xml_qfn) or die; binmode($fh); local $/; } ...Try to fix problems with unescaped characters... ...Pass $encoded_xml to parser...