Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

More efficient use of HTML::TokeParser::Simple

by henka (Novice)
on Jul 10, 2006 at 18:44 UTC ( #560199=perlquestion: print w/replies, xml ) Need Help??

henka has asked for the wisdom of the Perl Monks concerning the following question:

Hello folks. Thanks to some previous pointers from the monks, I am now using HTML::TokeParser::Simple to extract tags and their text from an HTML file. I would like to know if there is a more efficient way of doing the following (ie, instead of creating a new() HTML::TokeParser::Simple $p object each time, can you "rewind" the token position or something):
#!/usr/bin/perl use HTML::TokeParser::Simple; my $p = HTML::TokeParser::Simple->new( $ARGV[0]); #---------------------------------------- # get TITLE tag and value if ($p->get_tag("title")) { my $title = $p->get_trimmed_text; print "Title: $title\n"; } #---------------------------------------- # Now, from same file, get <H>eaders my $p = HTML::TokeParser::Simple->new( $ARGV[0]); while (my $token = $p->get_tag('h1')) { my $text = $p->get_trimmed_text("/h1"); print "H1: $text\n"; }

Replies are listed 'Best First'.
Re: More efficient use of HTML::TokeParser::Simple
by liverpole (Monsignor) on Jul 10, 2006 at 20:05 UTC
    Hi henka,

    I hadn't used HTML::TokeParser::Simple before, but I just tried it now, and it works fine for me even after removing the second call:

    my $p = HTML::TokeParser::Simple->new($ARGV[0]);

    So it seems logical that you would only instantiate the object once at the beginning.

    (Update:  Whoops ... not necessarily the case, as several wiser monks have gently instructed below.)

    A couple of points I'd mention, though.  First, if you add:

    use strict; use warnings;

    at the beginning of your code, you'll see that you've redefined $p.  (The strict and warnings pragmas are considered good programming practice, as they will catch a lot of things that might otherwise cause you problems -- it's a good idea to use them in all your programs).

    Secondly, it might be more helpful to the user (or even yourself!) if you give an error message or a syntax message in case no argument are passed to the program.  For example:

    use HTML::TokeParser::Simple; (my $url = shift) or die "syntax: html_test <filename>\n"; my $p = HTML::TokeParser::Simple->new($url);

    s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/
      I hadn't used HTML::TokeParser::Simple before, but I just tried it now, and it works fine for me even after removing the second call:
      my $p = HTML::TokeParser::Simple->new($ARGV[0]);
      Only if all the tags your want to capture next, come below the first one ("title"). It's very likely they do.

      Can you rewind a HTML::TokeParser::Simple? I don't think you can. But you can go through the tags one at a time, and check if it's one you're interested in, be it "title" or "h1". I wish it provided a way to scan for any of ("title", "h1"). Perhaps it will in the future, I think I'm going to supply a patch to that effect.

      Thanks liverpole - I always use strict/warnings - this was just a copy/paste of a test script.
Re: More efficient use of HTML::TokeParser::Simple
by GrandFather (Saint) on Jul 10, 2006 at 20:54 UTC

    If the HTML you are processing is modest in size then you might consider HTML::TreeBuilder which allows you to search for elements using various match criteria and may clean up code where you want to skip about the document.


    DWIM is Perl's answer to Gödel
      I poked around HTML::TreeBuilder, but my goodness, things are complicated. It may not seem like it to seasoned monks, but to a C programmer, the OO aspects and data structures of perl are, well, daunting. Gleaning how to do something as simple as the one I posted here from the perl module docs is almost always an excercise in frustration.

        Here's a trivial example that seems to do something like what you want and may be enough to get you started with TreeBuilder:

        use warnings; use strict; use HTML::TreeBuilder; my $html = do {local $/; <DATA>}; my $tree = HTML::TreeBuilder->new (); $tree->parse ($html); $tree->eof (); $tree->elementify(); my ($title) = $tree->find ('title'); my @h1 = $tree->find ('h1'); print $title->as_text (), "\n"; print $_->as_text (), "\n" for @h1; __DATA__ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <!-- Took this out for IE6ites "http://www.w3.org/TR/REC-html40/loose. +dtd" --> <html lang="en"> <head> <title>More efficient use of HTML::TokeParser::Simple perlquestion + id:560199</title> </head> <body> <h1>Header 1</h1> <p>First paragraph</p> <h1>Header 2</h1> <p>Second paragraph</p> <h2>Level 2 header 1</h2> </body> </html>

        Prints:

        More efficient use of HTML::TokeParser::Simple perlquestion id:560199 Header 1 Header 2

        DWIM is Perl's answer to Gödel
Re: More efficient use of HTML::TokeParser::Simple
by Ieronim (Friar) on Jul 10, 2006 at 20:41 UTC
    HTML::TokeParser (and HTML::TokeParser::Simple, too, as its subclass) fetches all tags in the document one-by-one. So, as the <TITLE> tag is always before any <H*> tags, the creation of new parser object makes little sense.
Re: More efficient use of HTML::TokeParser::Simple
by Khen1950fx (Canon) on Jul 10, 2006 at 20:59 UTC
    You might want to try HTML::PullParser as an alternative. You associate an argspec with an event that you want. Then this will return the next token found:  $token = $p->get_token Then try $p->unget_token( @tokens ) This will push back the tokens, but they'll be returned the next time $p->get_token is called. I hope this helps you.
Re: More efficient use of HTML::TokeParser::Simple
by graff (Chancellor) on Jul 11, 2006 at 03:58 UTC
    Have you considered an approach like this (this strategy is described in the HTML::TokeParser::Simple man page):
    use HTML::TokeParser::Simple; my $p = HTML::TokeParser::Simple->new( $ARGV[0]); my $state = ''; my %content; while ( my $tkn = $p->get_token ) { if ( $tkn->is_start_tag( 'title' )) { $state = "inTitle"; } elsif ( $tkn->is_start_tag( 'h1' )) { $state = "inH1"; } elsif ( $tkn->is_end_tag( 'title' ) or $tkn->is_end_tag( 'h1' )) { $state = ''; } elsif ( $tkn->is_text( ) and $state ) { $content{$state} .= $tkn->as_is; } } print "Title: $content{inTitle}\n"; print "H1: $content{inH1}\n";
    Depending on what your input data looks like (and how many elements besides "title" and "h1" you want to handle), this might not do exactly what you want, but I hope it will put you on the right path.
      Just by way of closing this post, and giving back a bit so that others in my position can search for it (and find the answer without bothering the murmuring monks):
      #!/usr/bin/perl use strict; use warnings; no warnings 'uninitialized'; use HTML::TokeParser::Simple; my $p = HTML::TokeParser::Simple->new($ARGV[0]); my $tag = ''; while (my $tkn = $p->get_token) { if ($tkn->is_start_tag('title')) { $tag = "TITLE"; } elsif ($tkn->is_start_tag('h1')) { $tag = "H1"; } elsif ($tkn->is_start_tag('h2')) { $tag = "H2"; } elsif ($tkn->is_start_tag('h3')) { $tag = "H3"; } elsif ($tkn->is_start_tag('h4')) { $tag = "H4"; } elsif ($tkn->is_start_tag('h5')) { $tag = "H5"; } elsif ($tkn->is_start_tag('h6')) { $tag = "H6"; } elsif ($tkn->is_start_tag('b')) { $tag = "B"; } elsif ($tkn->is_start_tag('i')) { $tag = "I"; } elsif ($tkn->is_start_tag('u')) { $tag = "U"; } elsif ($tkn->is_start_tag('a')) { $tag = "A"; } elsif ($tkn->is_start_tag('img')) { $tag = "IMG"; } elsif ($tkn->is_start_tag('meta')) { $tag = "META"; } elsif ( $tkn->is_end_tag('title') || $tkn->is_end_tag('h1') || $tkn->is_end_tag('h2') || $tkn->is_end_tag('h3') || $tkn->is_end_tag('h4') || $tkn->is_end_tag('h5') || $tkn->is_end_tag('h6') || $tkn->is_end_tag('b') || $tkn->is_end_tag('i') || $tkn->is_end_tag('u') || $tkn->is_end_tag('a') || $tkn->is_end_tag('img') || $tkn->is_end_tag('meta') ) { $tag = ''; } elsif ($tkn->is_text() && $tag && $tag ne 'META') { print "TAG: $tag, VALUE: ".$tkn->as_is . "\n"; } if ($tag eq 'IMG' && $tkn->get_attr('alt')) { print "TAG: $tag, ALT VALUE: ". $tkn->get_attr('alt') . "\n"; } if ($tag eq 'META' && $tkn->get_attr('name') eq 'keywords' && $tkn->get_attr('content')) { print "META-TAG: $tag, KEYWORDS VALUE: ". $tkn->get_attr('cont +ent') . "\n"; } if ($tag eq 'META' && $tkn->get_attr('name') eq 'description' && $tkn->get_attr('content')) { print "META-TAG: $tag, DESCRIPTION VALUE: ". $tkn->get_attr('c +ontent') . "\n"; } }
        The $t->is_start_tag can take a regex which could be useful in this case.

        Also, the title and meta tags appear in the head section of the html and all the others appear later in the body section. In these cases I use two loops.

        #!/usr/bin/perl use strict; use warnings; use HTML::TokeParser::Simple; my $html = join '', <DATA>; my $p = HTML::TokeParser::Simple->new(\$html); my ($title, $content, $keywords); while (my $t = $p->get_token){ last if $t->is_start_tag('body'); $title = $p->get_trimmed_text('/title') if $t->is_start_tag('title'); $content = $t->get_attr('content') if $t->is_start_tag('meta') and $t->get_attr('name') and $t->get_attr('name') eq 'Description'; $keywords = $t->get_attr('content') if $t->is_start_tag('meta') and $t->get_attr('name') and $t->get_attr('name') eq 'Keywords'; } print "title: $title\n"; print "content: $content\n"; my $tag; while (my $t = $p->get_token) { $tag = $t->get_tag if $t->is_start_tag(qr/^h[123456]|[biua]$/); if ($t->is_start_tag('img') and $t->get_attr('alt')){ my $attr = $t->get_attr('alt'); print "img attr: $attr\n"; $tag = ''; } elsif ($tag and $t->is_text){ my $txt = $t->as_is; print "$tag: $txt\n"; $tag = ''; } } __DATA__ <html> <head> <title>henka's test page</title> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1 +" /> <meta name="Description" content="the glories of HTML::TokeParser::Sim +ple" /> <meta name="keywords" content="one two three four five six seven eight + nine ten" /> <meta name="robots" content="noindex" /> <link rel="stylesheet" type="text/css" href="cwi.css" /> </head> <body> <h1>header one</h1> <h2>header two</h2> <h3>header three</h3> <h4>header four</h4> <h5>header five</h5> <h6>header siz</h6> <p>p tag paragraph</p> <p>p tag containing <u>underline</u> and <b>bold</b> and a <a href="li +nk.html">link</a></p> <img alt="image alt text" src="my.gif"> </body> </html>
        output:
        ---------- Capture Output ---------- > "c:\perl\bin\perl.exe" monk06.pl title: henka's test page content: the glories of HTML::TokeParser::Simple h1: header one h2: header two h3: header three h4: header four h5: header five h6: header siz u: underline b: bold a: link img attr: image alt text > Terminated with exit code 0.
      hmm, this looks like the ticket - thanks graff.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://560199]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2021-10-27 20:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (94 votes). Check out past polls.

    Notices?