Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
PerlMonks a poetry scraper

by eweaverp (Scribe)
on Jul 15, 2004 at 19:42 UTC ( #374801=sourcecode: print w/replies, xml ) Need Help??
Category: Fun Stuff
Author/Contact Info
Description: Grabs the complete works of any poet on and sticks them in a text file (alphabatized). Includes _PAGEBREAK_ markers for your favorite word processor's find/replace command.

Quick and dirty.

Invoke with  ./ NumberOfPoetAccordingToPlagiaristCom OutputFilePrefix

i.e.: ./ 91 jeffers_robinson

use strict;
use warnings;

my $poet = $ARGV[0];
my $name = $ARGV[1];

if (!$poet || !$name) {
    print "No poet specified.";

my $flag = 1;
my $count = 1;
my %hash;

print "Going to retrieve poet number $poet, who you say is $name.\n";

while ($flag) {

    $flag = 0;

    if ($count > 1) {
        print "On page $count.\n";
        $poet = $ARGV[0] . "\/$count";

    unlink($name . "_poet.html");
    system("wget -q -O\'" . $name . "_poet.html\' \'http:\/\/plagiaris\/poetry\/poets\/" . $poet . "\/\'");
    open(FILE, $name . "_poet.html");

    my $line;
    $flag = 0;
    foreach $line (<FILE>) {
        if ($line =~ m/<li><a href=\"http\:\/\/plagiarist\.com\/poetry
+\/(\d+)\/\">/) {
            $flag = 1;
            print "           Getting poem number $1.\n";
            system("wget -q -O\'" . $name . "_temp.html\' \'http://pla" . $1 . "/\'");
            open(POEM, $name . "_temp.html");

                local $/ = undef;
                my $poem = <POEM>;
                my ($match) = $poem =~ m/<div id=\"poem\">(.*?)<\/div>
                my ($title) = $match =~ m/<title>(.*?)<\/title>/ms;
                $match =~ s/(?:<.*?>|\<!--|-->|\n\n\n)//mg;
                $match =~ s/^(?:Submitted by.*|    poem|    )$//mg;

                $hash{$title} .= $match;




    $count ++;

print "Writing it down...\n";

unlink($name . ".txt");
open(OUTPUT, ">$name" . ".txt");

my $title;

foreach $title (sort(keys(%hash))) {
    print "       $title\n";
    print OUTPUT "_PAGEBREAK_\n\n$hash{$title}\n";


unlink($name . "_temp.html");
unlink($name . "_poet.html");
Replies are listed 'Best First'.
Re: a poetry scraper
by nmcfarl (Pilgrim) on Jul 16, 2004 at 17:01 UTC

    A resource I didn't know existed until I had a tool, your script, to use it effectively . Cool.

    Along the lines of constructive criticism, the excessive use of '\' make the script a little hard to read at points. Use of q{} , qq{} but mainly alternate delimiters on your m// (that is going with m{} or m|| ) could make things a good bit more readable. See Regexp Quote Like Operators

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://374801]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (2)
As of 2020-09-27 10:47 GMT
Find Nodes?
    Voting Booth?
    If at first I donít succeed, I Ö

    Results (142 votes). Check out past polls.