http://qs321.pair.com?node_id=1227701

nysus has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to properly handle UTF-8 characters scraped from a web page using the Web::Scraper module. Here is some sample code that simulates a snippet from a utf-8 encoded web page with an ellipsis character:

use warnings; use strict; use utf8; use Data::Dumper qw(Dumper); use Web::Scraper::LibXML; my $scraper = scraper { process '//p[contains(@class, "myClass")]', 'paragraph' => 'TEXT'; }; print Dumper $scraper->scrape(\<<'END_HTML'); <p class="myClass">…</p> END_HTML __END__

This script outputs:

$VAR1 = { 'paragraph' => "\x{2026}" };

I've tried various methods of encoding the input and/or decoding the output of the print statement using different methods and modules but I'm having no luck figuring out how to properly handle these characters so the utf8 characters are generated as they appear in the original source code. As usual, character encoding is confusing the hell out of me. The Web::Scraper manual says "If you pass URI or HTTP::Response object, Web::Scraper will automatically guesses the encoding of the content by looking at Content-Type headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to scrape method." Can someone please point me in the right direction?

$PM = "Perl Monk's";
$MCF = "Most Clueless Friar Abbot Bishop Pontiff Deacon Curate Priest Vicar";
$nysus = $PM . ' ' . $MCF;
Click here if you love Perl Monks