Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Handling utf-8 characters when scraping

by nysus (Vicar)
on Dec 26, 2018 at 06:14 UTC ( #1227701=perlquestion: print w/replies, xml ) Need Help??

nysus has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to properly handle UTF-8 characters scraped from a web page using the Web::Scraper module. Here is some sample code that simulates a snippet from a utf-8 encoded web page with an ellipsis character:

use warnings; use strict; use utf8; use Data::Dumper qw(Dumper); use Web::Scraper::LibXML; my $scraper = scraper { process '//p[contains(@class, "myClass")]', 'paragraph' => 'TEXT'; }; print Dumper $scraper->scrape(\<<'END_HTML'); <p class="myClass"></p> END_HTML __END__

This script outputs:

$VAR1 = { 'paragraph' => "\x{2026}" };

I've tried various methods of encoding the input and/or decoding the output of the print statement using different methods and modules but I'm having no luck figuring out how to properly handle these characters so the utf8 characters are generated as they appear in the original source code. As usual, character encoding is confusing the hell out of me. The Web::Scraper manual says "If you pass URI or HTTP::Response object, Web::Scraper will automatically guesses the encoding of the content by looking at Content-Type headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to scrape method." Can someone please point me in the right direction?

$PM = "Perl Monk's";
$MCF = "Most Clueless Friar Abbot Bishop Pontiff Deacon Curate Priest Vicar";
$nysus = $PM . ' ' . $MCF;
Click here if you love Perl Monks

Replies are listed 'Best First'.
Re: Handling utf-8 characters when scraping
by haukex (Bishop) on Dec 26, 2018 at 08:35 UTC

    Assuming that your source file is encoded correctly in UTF-8, then the output you've shown is correct - \x{2026} is U+2026 HORIZONTAL ELLIPSIS. Could you show an SSCCE of code you're having trouble with?

      The code I'm having trouble with works just like the sample code above. I don't want the ellipsis getting to get encoded into \x{2026}.

      $PM = "Perl Monk's";
      $MCF = "Most Clueless Friar Abbot Bishop Pontiff Deacon Curate Priest Vicar";
      $nysus = $PM . ' ' . $MCF;
      Click here if you love Perl Monks

        I don't want the ellipsis getting to get encoded into \x{2026}.

        That's Data::Dumper doing that, and I don't think there's a way to turn it off. Data::Dump seems to be similar. Data::Printer does seem to do what you want:

        use warnings; use strict; use open qw/:std :utf8/; use Data::Printer { print_escapes=>1 }; my $str = "(\N{U+2026}\n)"; p $str;

        Gives me: "(\n)"

        ... although on the other hand, these modules are all debugging tools, and not really tools for generating consistently formatted output. For that, other formats are better - if you could explain the application, then perhaps we could make other suggestions.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1227701]
Approved by Athanasius
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (3)
As of 2020-11-25 03:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?