http://qs321.pair.com?node_id=11116573


in reply to Why a regex *really* isn't good enough for HTML and XML, even for "simple" tasks

Here's the solution using plain WWW::Mechanize. It fails the XHTML test, because (I think) it uses HTML::TokeParser, and somehow misparses the Six link:

#!/usr/bin/env perl use warnings; use strict; my $file = shift or die; print "##### WWW::Mechanize on $file #####\n"; my $html = do { open my $fh, '<', $file or die "$file: $!"; local $/; +<$fh> }; use WWW::Mechanize; my $mech = WWW::Mechanize->new(); $mech->update_html($html); my @links = $mech->links(); for my $link (@links) { print $link->url, "\t", $link->text, "\n"; }

Since HTML::TokeParser and HTML::Parser even live in the same distribution, I'll look at a pull request to change the parser type to the one that works.

Update: The pull request

  • Comment on Re: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
  • Download Code

Replies are listed 'Best First'.
Re^2: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
by haukex (Archbishop) on May 08, 2020 at 07:47 UTC

    Thank you, I've added this one to the Gist too! (and in the meantime I added a XML::LibXML solution for the XHTML as well)

    Since HTML::TokeParser and HTML::Parser even live in the same distribution, I'll look at a pull request to change the parser type to the one that works.

    It looks to me like HTML::TokeParser isa HTML::Parser, so I think it's probably possible to set the options required to parse the XHTML (marked_sections and xml_mode) - but it looks like WWW::Mechanize doesn't provide any way to set custom options on the parser.

Re^2: Why a regex *really* isn't good enough for HTML, even for "simple" tasks
by haukex (Archbishop) on May 14, 2020 at 06:53 UTC