Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much


by lshatzer (Friar)
on Apr 09, 2002 at 00:53 UTC ( #157614=sourcecode: print w/replies, xml ) Need Help??
Category: Web Stuff
Author/Contact Info lshatzer
Description: This will detect if it is a URL, file, or html and pass it to HTML::TokeParser, and returns the HTML::TokeParser object. (This was my first venture into inheritance.)
Updated: Changed a few things from Amoe's suggestions.
package HTML::TokeParser::Smart;
require 5.006;
use strict;
use warnings;

use Carp;
use LWP;
use base 'HTML::TokeParser';
our $VERSION = '0.2';

sub new {
  my $proto = shift;
  my $class = ref($proto) || $proto;
  my $url = shift;
  my $self;
  if (-e $url) {
    # It's a file!
    $self = HTML::TokeParser->new($url);
  elsif ($url =~ m/^https?|^ftp|^file/) {
    # It's a URL!
    my $browser = LWP::UserAgent->new;
    my $req = $browser->request(HTTP::Request->new(GET=>$url));
    croak "Unable to get webpage: $url ", $req->status_line unless $re
    $self = HTML::TokeParser->new($req->content_ref);
  elsif ($url =~ m/<[^>]+>/) {
    # It's HTML!
    $self = HTML::TokeParser->new(\$url);

  else {
    croak "'$url' is neither a valid URL, file, or HTML.";
  bless ($self, $class);
  return $self;

Replies are listed 'Best First'.
Re: HTML::TokeParser::Smart
by Amoe (Friar) on Apr 09, 2002 at 10:22 UTC

    Aww, I was hoping it would automagically figure out which bits of HTML I wanted... ;)

    Couple of points to make about this, though:

    • Saying use base 'HTML::TokeParser' is IMHO a lot cleaner than that BEGIN mess.
    • If your class user provides a URL with angle brackets in and forgets to URI-escape them, you'll pass it as HTML, which probably won't dwim. Perhaps there's a module you could use to check if a file's HTML?
    • Using LWP::Simple makes saying that middle bit a lot easier, and since you're not using any of the extra featues of UserAgent its use is redundant here.

    Actually, you could solve that middle problem quite neatly by just switching the order of the tests so that it went (file - url - html) rather than (file - html - url), since all URLs acceptable by LWP will have a protocol, and URIs in HTML must be wrapped in a tag.

      Thanks for your input, I also recived some suggestons from crazyinsomiac to take a look at URI::URL, since I knew my url checking was not the best, and I had planned on changing it.

      Your last bullet point, see my use.perl journal, it has some info from Sean Burke. Any other suggestions would be welcome.

      Also, a better name would be good, another comment from crazyinsomniac says the Smart part dosn't indicate what the module does, but does not have any suggestions...

      Some other ideas I had was HTML::TokeParser::URL, but it will also do files or html.

      On an unrelated note, this is my 42nd post, not a 'major' milestone, but I think it's cool.
        lshatzer, I am very interested to use this package since I desparately need to URL in conjunction with HTML::TokeParser.
        I am new to LWP stuff.
        Would you please give some pointer as to how I can use this package? Thank you

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://157614]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (6)
As of 2020-09-30 06:04 GMT
Find Nodes?
    Voting Booth?
    If at first I donít succeed, I Ö

    Results (160 votes). Check out past polls.