Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re: Why a regex *really* isn't good enough for HTML, even for "simple" tasks

by Corion (Pope)
on May 08, 2020 at 07:30 UTC ( #11116573=note: print w/replies, xml ) Need Help??


in reply to Why a regex *really* isn't good enough for HTML and XML, even for "simple" tasks

Here's the solution using plain WWW::Mechanize. It fails the XHTML test, because (I think) it uses HTML::TokeParser, and somehow misparses the Six link:

#!/usr/bin/env perl use warnings; use strict; my $file = shift or die; print "##### WWW::Mechanize on $file #####\n"; my $html = do { open my $fh, '<', $file or die "$file: $!"; local $/; +<$fh> }; use WWW::Mechanize; my $mech = WWW::Mechanize->new(); $mech->update_html($html); my @links = $mech->links(); for my $link (@links) { print $link->url, "\t", $link->text, "\n"; }

Since HTML::TokeParser and HTML::Parser even live in the same distribution, I'll look at a pull request to change the parser type to the one that works.

Update: The pull request

  • Comment on Re: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
  • Download Code

Replies are listed 'Best First'.
Re^2: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
by haukex (Bishop) on May 08, 2020 at 07:47 UTC

    Thank you, I've added this one to the Gist too! (and in the meantime I added a XML::LibXML solution for the XHTML as well)

    Since HTML::TokeParser and HTML::Parser even live in the same distribution, I'll look at a pull request to change the parser type to the one that works.

    It looks to me like HTML::TokeParser isa HTML::Parser, so I think it's probably possible to set the options required to parse the XHTML (marked_sections and xml_mode) - but it looks like WWW::Mechanize doesn't provide any way to set custom options on the parser.

Re^2: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
by haukex (Bishop) on May 14, 2020 at 06:53 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11116573]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (3)
As of 2020-11-27 05:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?