When fetching a web page containing UTF8-encoded content from a server that doesn't say that it's sending out UTF8 via a response header, I'm getting this warning with LWP::Useragent on a simple
get():
Parsing of undecoded UTF-8 will give garbage
when decoding entities at .../LWP/Protocol.pm line xx.
The cause seems to be HTML::Parser (or better HTML::HeadParser) which assumes non-UTF8, but sees something
that looks like UTF8.
It's a warning, so I guess I could just suppress it, but I wanted to know if you guys had seen this before and have figured out an elegant way to deal with it.
As explained here, parse_head() is necessary to deal with some oddball webservers, otherwise I could turn it off in LWP::UserAgent's constructor.