Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Extracting information from the SETI@Home PM group

by Rhose (Priest)
on Dec 11, 2001 at 20:51 UTC ( #130970=sourcecode: print w/replies, xml ) Need Help??
Category: HTML Utility
Author/Contact Info Rhose
Description: I had never used the HTML::TableExtract module, so I created this script as a learning experience. As normal, if anyone has suggestions on things which I could do better, I would love to read them.
#!/usr/bin/perl -w
use strict;
#-- Script:
#-- Purpose:   Displays the information for the SETI@Home PM group
#-- Author:    Robert(Bob) Smith
#-- Date:      December 11, 2001
#-- Wish List: Add error handling if the requested HTML page is not re
#-- Rev Hist:  00.00.a 2001-12-11 rws  Initial version
#-- Notes:     This script was created as a learning example for HTML:

#-- Use modules
use HTML::TableExtract;
use HTTP::Request::Common;
use LWP::UserAgent;

#-- Define constants
use constant VERSION                   => '00.00.a';

use constant FIELD_DELIM               => ',';
use constant SETI_URL                  => 'http://setiathome.ssl.berke';

use constant ERR                       =>
    'ok'                               => 0,

#-- Define variables
my $gExtractedTable;                   #  Table extracted from the HTM
my $gHTMLPage;                         #  Retrieved HTML page
my $gName;                             #  Member's name
my $gRank;                             #  Member's ranking
my $gRow;                              #  Pointer to rows in extracted
+ tables
my $gTable;                            #  Pointer to extracted tables
my $gUserAgent;                        #  LWP::UserAgent

#-- Retrieve the HTML page
$gUserAgent = LWP::UserAgent->new;
$gHTMLPage = $gUserAgent->request(GET SETI_URL);

#-- Extract the table
#-- Note: TableExtract will handle tables nested within tables (outerm
+ost table is depth==0)
#--       as well as multiple tables within the same HTML document (fi
+rst table is count==0).
#--       Since the information I wish to extract is not nested, depth
+ will be 0, and since
#--       it is the second table on the page (the first table is the g
+roup description,
#--       web site, number of members, etc...,) the count will be 1.
$gExtractedTable = HTML::TableExtract->new(depth => 0, count => 1);

#-- Display information
foreach $gTable ($gExtractedTable->table_states)
  foreach $gRow ($gTable->rows)

    #-- Print data row
    if ($$gRow[0]=~/^(\d+)\)\s*/)
      ($gRank, $gName)=($1,$');
      $gName=$` if $gName=~/[\s\n]+$/;
      $$gRow[2]=$' if $$gRow[2]=~/^\s+/;

      print $gRank, FIELD_DELIM,
            $gName, FIELD_DELIM,
            $$gRow[1], FIELD_DELIM,
            $$gRow[2], FIELD_DELIM,
            $$gRow[3], "\n";

    #-- Print header row
      $$gRow[1]=~tr/\n/ /;
      $$gRow[3]=~tr/\n/ /;

      print 'Rank', FIELD_DELIM,
            $$gRow[0], FIELD_DELIM,
            $$gRow[1], FIELD_DELIM,
            $$gRow[2], FIELD_DELIM,
            $$gRow[3], "\n";

#-- Exit

#-- End of script
Replies are listed 'Best First'.
Re: Extracting information from the SETI@Home PM group
by djw (Vicar) on Dec 17, 2001 at 20:20 UTC
    This is pretty cool.

    I have never tried HTML::TableExtract but now that I know about it, I'm going to give it a try. I have done something similar for my user stats here:, but I use LWP::Simple to grab the page, and HTML::TreeBuilder to strip the html out (then regex's to grab the data I want).

    Something else to note is that the perlmonks SETI page has a few table entries that are blank. In the html 'code', you can see they use the ' & n b s p ; ' code for a single space in each case (user #17 and user #80). When I run your code on my Win2k box, I get an 'a' with an inflection symbol (or whatever its called) above it. On my linux box I get a blank entry as you would expect. I mention this just as a head's up in case you didn't notice...

    Cool use of the module ++.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://130970]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2020-09-23 17:18 GMT
Find Nodes?
    Voting Booth?
    If at first I donít succeed, I Ö

    Results (132 votes). Check out past polls.