Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Slashdot headlines

by moxliukas (Curate)
on May 29, 2002 at 14:27 UTC ( #170090=sourcecode: print w/replies, xml ) Need Help??
Category: HTML Utilities
Author/Contact Info moxliukas <moxliukas AT delfi DOT lt>
Description: Well, I'm only a beginner in Perl, but I have whacked up a script that extracts headlines from Slashdot. Yes, it is very probably buggy and definately one can come up with a more elegant solution. It uses standard modules from CPAN, such as LWP::UserAgent, HTTP::Request and HTML::TreeBuilder.
use HTTP::Request;
use LWP::UserAgent;
use HTML::TreeBuilder;

$ua = LWP::UserAgent->new;
$request = HTTP::Request->new(GET => '');
$resp = $ua->request($request);

$tree = HTML::TreeBuilder->new;

my @titles;
my @authors;
my @dept;

foreach my $h ($tree->look_down('_tag', 'font', 
    sub {
      return unless ($_[0]->attr('face') and $_[0]->attr('size') and $
      my @c = $_[0]->content_list;
      return unless ref $c[0];
      my @d = $c[0]->content_list;
      return 1 if not ref $d[0];
      $#c == 0 and $c[0]->tag eq 'b' and $d[0]->tag ne 'map';
  )) {
  push @titles, $h->as_text;
foreach my $h ($tree->look_down('_tag', 'b', 
    sub {
      my @c = $_[0]->content_list;
      @c == 3 and ref $c[1] and $c[1]->tag eq 'a' and $c[0] =~ /^Poste
+d by/;
  )) {
  push @authors, $h->as_text;
foreach my $h ($tree->look_down('_tag', 'b', 
    sub {
      my @c = $_[0]->content_list;
      @c == 1 and $c[0] =~ /^from the .* dept\.$/;
  )) {
  push @dept, $h->as_text;

for($i=0; $i<10; $i++) {
  print "$titles[$i]\n";
  print "$authors[$i]\n";
  print "$dept[$i]\n\n";

$tree = $tree->delete;
Replies are listed 'Best First'.
•Re: Slashdot headlines
by merlyn (Sage) on May 29, 2002 at 18:00 UTC
    Here's a slightly simpler version that I whipped up in a few minutes, requiring XML::RSS and Template as well as LWP:
    #!/usr/bin/perl -w use strict; my $SLASHCACHE = "/Users/merlyn/slashdot.rdf"; use LWP::Simple; use Template; mirror("", $SLASHCACHE); my $t = Template->new; $t->process(\*DATA, { slashcache => $SLASHCACHE} ) or die $t->error; __END__ === Here's the news: [% USE news = XML.RSS(slashcache); FOREACH item = news.items; LAST IF loop.count > 10; -%] [% loop.count %]: [% item.title %] at [% %] [% END -%] ===

    -- Randal L. Schwartz, Perl hacker

      While merlyn's solution works well, I prefer a more XML native approach. The RDF file should be valid XML, in which case you only need apply an XSL Transformation to the file once you have it, to get valid XHTML, WML or whatever out.

      This is the basis of XML::RSS::Tools, a module I've been developing. Here is an example of the usage:

      #!/usr/bin/perl -w use strict; use XML::RSS::Tools; my $site = shift; my $xsl = shift; my $rss = XML::RSS::Tools->new; $rss->rss_uri($site) or die $rss->as_string('error'); $rss->xsl_file($xsl) or die $rss->as_string('error'); if ($rss->transform) { print $rss->as_string };

      There are however problems:

      • XML::RSS module has some defects
      • Perl before 5.7.x isn't exactly the best for working with Unicode
      • RSS feeds often are not valid XML - How do I clean RSS feeds to make them usable?
      • My module still has bugs in... :-(
      • UPDATE: Versions later than 0.05 have less bugs, and partially get round the defect in XML::RSS and dirty feeds

      My humble 2p

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: sourcecode [id://170090]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2022-05-27 22:27 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (98 votes). Check out past polls.