TokeParser and newlines

SilverSpring8 has asked for the wisdom of the Perl Monks concerning the following question:

O wise monks:

Noob (me) wants to extract a sports schedule from a website. Here's my code:

#!/usr/bin/perl -w
  
use strict;
use LWP::Simple;
use HTML::TokeParser;

my $url = 'http://www.georgiastatesports.com/SportSelect.dbml?DB_OEM_I
+D=12700&SPID=5671&SPSID=53628';
my $raw = get($url);
my $stream = HTML::TokeParser->new(\$raw)
  or die "$url $!";

while(my $token = $stream->get_token() ) {
  my $ttype = shift @{ $token };
  if ($ttype eq "S") {
    my($tag, $attr, $attrseq, $rawtxt) = @{ $token };
    next until ($tag eq "table");
    my $class = $attr->{'class'};
    if ($class eq "ScheduleTable") {
      my $sched = $stream->get_trimmed_text("/$tag");
      print "$sched\n";
    } # class eq "ScheduleTable"
  } # $ttype eq "S"
} # while
[download]

What I hoped to return:

Date Opponent Location Time (ET) Results Overall CAA Media
Thu, Aug 30 South Carolina State Georgia Dome 7:30 PM
Sat, Sep 08 Tennessee at Knoxville, Tenn. 4:00 PM PPV
Sat, Sep 15 UTSA Georgia Dome 6:00 PM
Sat, Sep 22 Richmond * Georgia Dome 3:30 PM
Sat, Sep 29 William & Mary * at Williamsburg, Va. 7:00 PM CSS-TV
Sat, Oct 06 New Hampshire * Georgia Dome 3:30 PM
Sat, Oct 13 Rhode Island * at Kingston, R.I. 1:00 PM
Sat, Oct 20 Villanova (Homecoming) * Georgia Dome 3:30 PM
Sat, Oct 27 James Madison * at Harrisonburg, Va. 3:30 PM
Sat, Nov 03 Old Dominion * Georgia Dome 3:30 PM
Sat, Nov 10 Maine * at Orono, Maine 2:00 PM
[download]

What I got:

Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Date Opponent Location Time (ET) Results Overall CAA Media ? Thu, Aug 
+30 ?South Carolina State Georgia Dome ? 7:30 PM ? Sat, Sep 08 ?Tennes
+see at Knoxville, Tenn. ? 4:00 PM PPV ? Sat, Sep 15 ?UTSA Georgia Dom
+e ? 6:00 PM ? Sat, Sep 22 ?Richmond * Georgia Dome ? 3:30 PM ? Sat, S
+ep 29 ?William & Mary * at Williamsburg, Va. ? 7:00 PM CSS-TV ? Sat, 
+Oct 06 ?New Hampshire * Georgia Dome ? 3:30 PM ? Sat, Oct 13 ?Rhode I
+sland * at Kingston, R.I. ? 1:00 PM ? Sat, Oct 20 ?Villanova (Homecom
+ing) * Georgia Dome ? 3:30 PM ? Sat, Oct 27 ?James Madison * at Harri
+sonburg, Va. ? 3:30 PM ? Sat, Nov 03 ?Old Dominion * Georgia Dome ? 3
+:30 PM ? Sat, Nov 10 ?Maine * at Orono, Maine ? 2:00 PM
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
Use of uninitialized value $class in string eq at ./gsu2.pl line 18.
[download]

A hexdump where the newlines should be shows a character sequence of \0x20\0xA0\0x20. I tried

local $/ = \0xA0;

between get_trimmed_text() and print, without any apparent effect.

How would I get newlines before each date and handle the "uninitialized" warnings? I await your wisdom.

Comment on TokeParser and newlines Select or Download Code

Replies are listed 'Best First'.
Re: TokeParser and newlines by suaveant (Parson) on Aug 02, 2012 at 00:27 UTC
Well, this isn't exactly right but it'll move you along, don't have time to continue. Probably finding a more comprehensive parser as previously suggested would help #!/usr/bin/perl -w use strict; use LWP::Simple; use HTML::TokeParser; my $url = 'http://www.georgiastatesports.com/SportSelect.dbml?DB_OEM_I +D=12700&SPID=5671&SPSID=53628'; my $raw = get($url); my $stream = HTML::TokeParser->new(\$raw) or die "$url $!"; while(my $token = $stream->get_token() ) { my $ttype = shift @{ $token }; if ($ttype eq "S") { my($tag, $attr, $attrseq, $rawtxt) = @{ $token }; next until ($tag eq "table"); my $class = $attr->{'class'}; if ($class eq "ScheduleTable") { while($stream->get_tag('tr')) {# my $sched = $stream->get_trimmed_text('/tr'); print "$sched\n"; } } # class eq "ScheduleTable" } # $ttype eq "S" } # while [download] - Ant - Some of my best work - (1 2 3)	[reply] [d/l]
Re: TokeParser and newlines by Anonymous Monk on Aug 01, 2012 at 22:33 UTC
Well, first you'd ditch TokeParser for TreeBuilder::Xpath, then run your html file through htmltreexpather.pl and you're done :) see examples and walkthroughs: Parsing HTML / Re^4: Parsing HTML, A regex question , NASA's Astronomy Picture of the Day / Re: NASA's Astronomy Picture of the Day ....	[reply]
Re^2: TokeParser and newlines by SilverSpring8 (Initiate) on Aug 02, 2012 at 00:09 UTC
Thanks for the advice and the links. I'll check it out!	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks