Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

More efficient use of HTML::TokeParser::Simple

by henka (Novice)
on Jul 10, 2006 at 18:44 UTC ( [id://560199]=perlquestion: print w/replies, xml ) Need Help??

henka has asked for the wisdom of the Perl Monks concerning the following question:

Hello folks. Thanks to some previous pointers from the monks, I am now using HTML::TokeParser::Simple to extract tags and their text from an HTML file. I would like to know if there is a more efficient way of doing the following (ie, instead of creating a new() HTML::TokeParser::Simple $p object each time, can you "rewind" the token position or something): 
#!/usr/bin/perl

use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $ARGV[0]);

#----------------------------------------
# get TITLE tag and value
if ($p->get_tag("title")) {
    my $title = $p->get_trimmed_text;
    print "Title: $title\n";
}

#----------------------------------------
# Now, from same file, get <H>eaders
my $p = HTML::TokeParser::Simple->new( $ARGV[0]);
while (my $token = $p->get_tag('h1'))
{
    my $text = $p->get_trimmed_text("/h1");
    print "H1: $text\n";
}

[download]

Replies are listed 'Best First'.
Re: More efficient use of HTML::TokeParser::Simple
by liverpole (Monsignor) on Jul 10, 2006 at 20:05 UTC
    Hi henka,

    I hadn't used HTML::TokeParser::Simple before, but I just tried it now, and it works fine for me even after removing the second call: 

    my $p = HTML::TokeParser::Simple->new($ARGV[0]);

    [download]

    So it seems logical that you would only instantiate the object once at the beginning.

    (Update:  Whoops ... not necessarily the case, as several wiser monks have gently instructed below.)

    A couple of points I'd mention, though.  First, if you add: 

    use strict;
use warnings;

    [download]

    at the beginning of your code, you'll see that you've redefined $p.  (The strict and warnings pragmas are considered good programming practice, as they will catch a lot of things that might otherwise cause you problems -- it's a good idea to use them in all your programs).

    Secondly, it might be more helpful to the user (or even yourself!) if you give an error message or a syntax message in case no argument are passed to the program.  For example: 

    use HTML::TokeParser::Simple;
(my $url = shift) or die "syntax:  html_test  <filename>\n";
my $p = HTML::TokeParser::Simple->new($url);

    [download]
    s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/
[reply]
[d/l]
[select]
      I hadn't used HTML::TokeParser::Simple before, but I just tried it now, and it works fine for me even after removing the second call: 
      my $p = HTML::TokeParser::Simple->new($ARGV[0]);

      [download]
      Only if all the tags your want to capture next, come below the first one ("title"). It's very likely they do.

      Can you rewind a HTML::TokeParser::Simple? I don't think you can. But you can go through the tags one at a time, and check if it's one you're interested in, be it "title" or "h1". I wish it provided a way to scan for any of ("title", "h1"). Perhaps it will in the future, I think I'm going to supply a patch to that effect.

[reply]
[d/l]
[select]
      Thanks liverpole - I always use strict/warnings - this was just a copy/paste of a test script.
[reply]
Re: More efficient use of HTML::TokeParser::Simple
by GrandFather (Saint) on Jul 10, 2006 at 20:54 UTC

    If the HTML you are processing is modest in size then you might consider HTML::TreeBuilder which allows you to search for elements using various match criteria and may clean up code where you want to skip about the document.

    DWIM is Perl's answer to Gödel
[reply]
      I poked around HTML::TreeBuilder, but my goodness, things are complicated. It may not seem like it to seasoned monks, but to a C programmer, the OO aspects and data structures of perl are, well, daunting. Gleaning how to do something as simple as the one I posted here from the perl module docs is almost always an excercise in frustration.
[reply]

        Here's a trivial example that seems to do something like what you want and may be enough to get you started with TreeBuilder:

        use warnings;
use strict;
use HTML::TreeBuilder;

my $html = do {local $/; <DATA>};
my $tree = HTML::TreeBuilder->new ();

$tree->parse ($html);
$tree->eof ();
$tree->elementify();

my ($title) = $tree->find ('title');
my @h1 = $tree->find ('h1');

print $title->as_text (), "\n";
print $_->as_text (), "\n" for @h1;

__DATA__
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<!-- Took this out for IE6ites "http://www.w3.org/TR/REC-html40/loose.
+dtd" -->
<html lang="en">
  <head>
    <title>More efficient use of HTML::TokeParser::Simple perlquestion
+ id:560199</title>
  </head>
  <body>
  <h1>Header 1</h1>
  <p>First paragraph</p>
  <h1>Header 2</h1>
  <p>Second paragraph</p>
  <h2>Level 2 header 1</h2>
  </body>    
</html>

        [download]

        Prints:

        More efficient use of HTML::TokeParser::Simple perlquestion id:560199
Header 1
Header 2

        [download]
        DWIM is Perl's answer to Gödel
[reply]
[d/l]
[select]
Re: More efficient use of HTML::TokeParser::Simple
by Ieronim (Friar) on Jul 10, 2006 at 20:41 UTC
    HTML::TokeParser (and HTML::TokeParser::Simple, too, as its subclass) fetches all tags in the document one-by-one. So, as the <TITLE> tag is always before any <H*> tags, the creation of new parser object makes little sense.
[reply]
[d/l]
[select]
Re: More efficient use of HTML::TokeParser::Simple
by Khen1950fx (Canon) on Jul 10, 2006 at 20:59 UTC
    You might want to try HTML::PullParser as an alternative. You associate an argspec with an event that you want. Then this will return the next token found:  $token = $p->get_token Then try $p->unget_token( @tokens ) This will push back the tokens, but they'll be returned the next time $p->get_token is called. I hope this helps you.
[reply]
[d/l]
[select]
Re: More efficient use of HTML::TokeParser::Simple
by graff (Chancellor) on Jul 11, 2006 at 03:58 UTC
    Have you considered an approach like this (this strategy is described in the HTML::TokeParser::Simple man page): 
    use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $ARGV[0]);

my $state = '';
my %content;

while ( my $tkn = $p->get_token ) {
    if ( $tkn->is_start_tag( 'title' )) {
        $state = "inTitle";
    }
    elsif ( $tkn->is_start_tag( 'h1' )) {
        $state = "inH1";
    }
    elsif ( $tkn->is_end_tag( 'title' ) or
            $tkn->is_end_tag( 'h1' )) {
        $state = '';
    }
    elsif ( $tkn->is_text( ) and $state ) {
        $content{$state} .= $tkn->as_is;
    }
}

print "Title: $content{inTitle}\n";
print "H1: $content{inH1}\n";

    [download]
    Depending on what your input data looks like (and how many elements besides "title" and "h1" you want to handle), this might not do exactly what you want, but I hope it will put you on the right path.
[reply]
[d/l]
      Just by way of closing this post, and giving back a bit so that others in my position can search for it (and find the answer without bothering the murmuring monks): 
      #!/usr/bin/perl

use strict;
use warnings;
no warnings 'uninitialized';
use HTML::TokeParser::Simple;

my $p = HTML::TokeParser::Simple->new($ARGV[0]);
my $tag = '';

while (my $tkn = $p->get_token)
{
    if ($tkn->is_start_tag('title')) { $tag = "TITLE"; }

    elsif ($tkn->is_start_tag('h1')) { $tag = "H1"; }

    elsif ($tkn->is_start_tag('h2')) { $tag = "H2"; }

    elsif ($tkn->is_start_tag('h3')) { $tag = "H3"; }

    elsif ($tkn->is_start_tag('h4')) { $tag = "H4"; }

    elsif ($tkn->is_start_tag('h5')) { $tag = "H5"; }

    elsif ($tkn->is_start_tag('h6')) { $tag = "H6"; }

    elsif ($tkn->is_start_tag('b')) { $tag = "B"; }

    elsif ($tkn->is_start_tag('i')) { $tag = "I"; }

    elsif ($tkn->is_start_tag('u')) { $tag = "U"; }

    elsif ($tkn->is_start_tag('a')) { $tag = "A"; }

    elsif ($tkn->is_start_tag('img')) { $tag = "IMG"; }

    elsif ($tkn->is_start_tag('meta')) { $tag = "META"; }

    elsif (
        $tkn->is_end_tag('title') ||
        $tkn->is_end_tag('h1') ||
        $tkn->is_end_tag('h2') ||
        $tkn->is_end_tag('h3') ||
        $tkn->is_end_tag('h4') ||
        $tkn->is_end_tag('h5') ||
        $tkn->is_end_tag('h6') ||
        $tkn->is_end_tag('b')  ||
        $tkn->is_end_tag('i')  ||
        $tkn->is_end_tag('u')  ||
        $tkn->is_end_tag('a')  ||
        $tkn->is_end_tag('img') ||
        $tkn->is_end_tag('meta')
        ) { $tag = ''; }

    elsif ($tkn->is_text() && $tag && $tag ne 'META')
    {
        print "TAG: $tag, VALUE: ".$tkn->as_is . "\n";
    }

    if ($tag eq 'IMG' && $tkn->get_attr('alt'))
    {
        print "TAG: $tag, ALT VALUE: ". $tkn->get_attr('alt') . "\n";
    }

    if ($tag eq 'META' &&
        $tkn->get_attr('name') eq 'keywords' &&
        $tkn->get_attr('content'))
    {
        print "META-TAG: $tag, KEYWORDS VALUE: ". $tkn->get_attr('cont
+ent') . "\n";
    }

    if ($tag eq 'META' &&
        $tkn->get_attr('name') eq 'description' &&
        $tkn->get_attr('content'))
    {
        print "META-TAG: $tag, DESCRIPTION VALUE: ". $tkn->get_attr('c
+ontent') . "\n";
    }
}

      [download]
[reply]
[d/l]
        The $t->is_start_tag can take a regex which could be useful in this case.

        Also, the title and meta tags appear in the head section of the html and all the others appear later in the body section. In these cases I use two loops.

        #!/usr/bin/perl

use strict;
use warnings;
use HTML::TokeParser::Simple;

my $html = join '', <DATA>;

my $p = HTML::TokeParser::Simple->new(\$html);

my ($title, $content, $keywords);

while (my $t = $p->get_token){
  
  last if $t->is_start_tag('body');
  
  $title = $p->get_trimmed_text('/title') 
    if $t->is_start_tag('title');

  $content = $t->get_attr('content') if 
    $t->is_start_tag('meta') and
    $t->get_attr('name') and
    $t->get_attr('name') eq 'Description';
    
  $keywords = $t->get_attr('content') if 
    $t->is_start_tag('meta') and
    $t->get_attr('name') and
    $t->get_attr('name') eq 'Keywords';
    
}

print "title: $title\n";
print "content: $content\n";

my $tag;

while (my $t = $p->get_token) {

  $tag = $t->get_tag
    if $t->is_start_tag(qr/^h[123456]|[biua]$/);

  if ($t->is_start_tag('img') and $t->get_attr('alt')){ 
    my $attr = $t->get_attr('alt');
    print "img attr: $attr\n";
    $tag = '';
  }
  elsif ($tag and $t->is_text){
    my $txt = $t->as_is;      
    print "$tag: $txt\n";
    $tag = '';
  }
  
}

__DATA__
<html>
<head>
<title>henka's test page</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1
+" />
<meta name="Description" content="the glories of HTML::TokeParser::Sim
+ple" />
<meta name="keywords" content="one two three four five six seven eight
+ nine ten" />
<meta name="robots" content="noindex" />
<link rel="stylesheet" type="text/css" href="cwi.css" />
</head>

<body>
<h1>header one</h1>
<h2>header two</h2>
<h3>header three</h3>
<h4>header four</h4>
<h5>header five</h5>
<h6>header siz</h6>

<p>p tag paragraph</p>
<p>p tag containing <u>underline</u> and <b>bold</b> and a <a href="li
+nk.html">link</a></p>
<img alt="image alt text" src="my.gif">

</body>
</html>

        [download]
        output: 
        ---------- Capture Output ----------
> "c:\perl\bin\perl.exe" monk06.pl
title: henka's test page
content: the glories of HTML::TokeParser::Simple
h1: header one
h2: header two
h3: header three
h4: header four
h5: header five
h6: header siz
u: underline
b: bold
a: link
img attr: image alt text
> Terminated with exit code 0.

        [download]
[reply]
[d/l]
[select]
      hmm, this looks like the ticket - thanks graff.
[reply]

Back to Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: perlquestion [id://560199]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others lurking in the Monastery: (1)
As of 2024-04-25 00:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found