Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: More efficient use of HTML::TokeParser::Simple

by graff (Chancellor)
on Jul 11, 2006 at 03:58 UTC ( [id://560300]=note: print w/replies, xml ) Need Help??


in reply to More efficient use of HTML::TokeParser::Simple

Have you considered an approach like this (this strategy is described in the HTML::TokeParser::Simple man page): 
use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $ARGV[0]);

my $state = '';
my %content;

while ( my $tkn = $p->get_token ) {
    if ( $tkn->is_start_tag( 'title' )) {
        $state = "inTitle";
    }
    elsif ( $tkn->is_start_tag( 'h1' )) {
        $state = "inH1";
    }
    elsif ( $tkn->is_end_tag( 'title' ) or
            $tkn->is_end_tag( 'h1' )) {
        $state = '';
    }
    elsif ( $tkn->is_text( ) and $state ) {
        $content{$state} .= $tkn->as_is;
    }
}

print "Title: $content{inTitle}\n";
print "H1: $content{inH1}\n";

[download]
Depending on what your input data looks like (and how many elements besides "title" and "h1" you want to handle), this might not do exactly what you want, but I hope it will put you on the right path.

Replies are listed 'Best First'.
Re^2: More efficient use of HTML::TokeParser::Simple
by henka (Novice) on Jul 11, 2006 at 09:21 UTC
    Just by way of closing this post, and giving back a bit so that others in my position can search for it (and find the answer without bothering the murmuring monks): 
    #!/usr/bin/perl

use strict;
use warnings;
no warnings 'uninitialized';
use HTML::TokeParser::Simple;

my $p = HTML::TokeParser::Simple->new($ARGV[0]);
my $tag = '';

while (my $tkn = $p->get_token)
{
    if ($tkn->is_start_tag('title')) { $tag = "TITLE"; }

    elsif ($tkn->is_start_tag('h1')) { $tag = "H1"; }

    elsif ($tkn->is_start_tag('h2')) { $tag = "H2"; }

    elsif ($tkn->is_start_tag('h3')) { $tag = "H3"; }

    elsif ($tkn->is_start_tag('h4')) { $tag = "H4"; }

    elsif ($tkn->is_start_tag('h5')) { $tag = "H5"; }

    elsif ($tkn->is_start_tag('h6')) { $tag = "H6"; }

    elsif ($tkn->is_start_tag('b')) { $tag = "B"; }

    elsif ($tkn->is_start_tag('i')) { $tag = "I"; }

    elsif ($tkn->is_start_tag('u')) { $tag = "U"; }

    elsif ($tkn->is_start_tag('a')) { $tag = "A"; }

    elsif ($tkn->is_start_tag('img')) { $tag = "IMG"; }

    elsif ($tkn->is_start_tag('meta')) { $tag = "META"; }

    elsif (
        $tkn->is_end_tag('title') ||
        $tkn->is_end_tag('h1') ||
        $tkn->is_end_tag('h2') ||
        $tkn->is_end_tag('h3') ||
        $tkn->is_end_tag('h4') ||
        $tkn->is_end_tag('h5') ||
        $tkn->is_end_tag('h6') ||
        $tkn->is_end_tag('b')  ||
        $tkn->is_end_tag('i')  ||
        $tkn->is_end_tag('u')  ||
        $tkn->is_end_tag('a')  ||
        $tkn->is_end_tag('img') ||
        $tkn->is_end_tag('meta')
        ) { $tag = ''; }

    elsif ($tkn->is_text() && $tag && $tag ne 'META')
    {
        print "TAG: $tag, VALUE: ".$tkn->as_is . "\n";
    }

    if ($tag eq 'IMG' && $tkn->get_attr('alt'))
    {
        print "TAG: $tag, ALT VALUE: ". $tkn->get_attr('alt') . "\n";
    }

    if ($tag eq 'META' &&
        $tkn->get_attr('name') eq 'keywords' &&
        $tkn->get_attr('content'))
    {
        print "META-TAG: $tag, KEYWORDS VALUE: ". $tkn->get_attr('cont
+ent') . "\n";
    }

    if ($tag eq 'META' &&
        $tkn->get_attr('name') eq 'description' &&
        $tkn->get_attr('content'))
    {
        print "META-TAG: $tag, DESCRIPTION VALUE: ". $tkn->get_attr('c
+ontent') . "\n";
    }
}

    [download]
[reply]
[d/l]
      The $t->is_start_tag can take a regex which could be useful in this case.

      Also, the title and meta tags appear in the head section of the html and all the others appear later in the body section. In these cases I use two loops.

      #!/usr/bin/perl

use strict;
use warnings;
use HTML::TokeParser::Simple;

my $html = join '', <DATA>;

my $p = HTML::TokeParser::Simple->new(\$html);

my ($title, $content, $keywords);

while (my $t = $p->get_token){
  
  last if $t->is_start_tag('body');
  
  $title = $p->get_trimmed_text('/title') 
    if $t->is_start_tag('title');

  $content = $t->get_attr('content') if 
    $t->is_start_tag('meta') and
    $t->get_attr('name') and
    $t->get_attr('name') eq 'Description';
    
  $keywords = $t->get_attr('content') if 
    $t->is_start_tag('meta') and
    $t->get_attr('name') and
    $t->get_attr('name') eq 'Keywords';
    
}

print "title: $title\n";
print "content: $content\n";

my $tag;

while (my $t = $p->get_token) {

  $tag = $t->get_tag
    if $t->is_start_tag(qr/^h[123456]|[biua]$/);

  if ($t->is_start_tag('img') and $t->get_attr('alt')){ 
    my $attr = $t->get_attr('alt');
    print "img attr: $attr\n";
    $tag = '';
  }
  elsif ($tag and $t->is_text){
    my $txt = $t->as_is;      
    print "$tag: $txt\n";
    $tag = '';
  }
  
}

__DATA__
<html>
<head>
<title>henka's test page</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1
+" />
<meta name="Description" content="the glories of HTML::TokeParser::Sim
+ple" />
<meta name="keywords" content="one two three four five six seven eight
+ nine ten" />
<meta name="robots" content="noindex" />
<link rel="stylesheet" type="text/css" href="cwi.css" />
</head>

<body>
<h1>header one</h1>
<h2>header two</h2>
<h3>header three</h3>
<h4>header four</h4>
<h5>header five</h5>
<h6>header siz</h6>

<p>p tag paragraph</p>
<p>p tag containing <u>underline</u> and <b>bold</b> and a <a href="li
+nk.html">link</a></p>
<img alt="image alt text" src="my.gif">

</body>
</html>

      [download]
      output: 
      ---------- Capture Output ----------
> "c:\perl\bin\perl.exe" monk06.pl
title: henka's test page
content: the glories of HTML::TokeParser::Simple
h1: header one
h2: header two
h3: header three
h4: header four
h5: header five
h6: header siz
u: underline
b: bold
a: link
img attr: image alt text
> Terminated with exit code 0.

      [download]
[reply]
[d/l]
[select]
Re^2: More efficient use of HTML::TokeParser::Simple
by henka (Novice) on Jul 11, 2006 at 06:46 UTC
    hmm, this looks like the ticket - thanks graff.
[reply]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://560300]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others having a coffee break in the Monastery: (4)
As of 2024-04-26 05:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found