Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

LWP spider

by Anfield (Initiate)
on Mar 20, 2002 at 19:18 UTC ( [id://153091]=perlquestion: print w/replies, xml ) Need Help??

Anfield has asked for the wisdom of the Perl Monks concerning the following question:

There was an LWP spider script on perl website like 2 years ago. I can not find it any more now. Does any one here have a similiar script? The script needed must be able to go deep at a pre-set value to fetch links. thanx in advance

Replies are listed 'Best First'.
Re: LWP spider
by vagnerr (Prior) on Mar 20, 2002 at 19:58 UTC
    When you install LWP a couple of perl scripts are installed at the same time called lwp-rget and lwp-request. lwp-rget is quite a nice spider program supporting basic auth and cookie files so that may be what you're looking for.

    ---If it doesn't fit use a bigger hammer
Re: LWP spider
by tachyon (Chancellor) on Mar 20, 2002 at 20:43 UTC

    Link Checker is an LWP NET::FTP HTML::TokeParser link checking spider I wrote that crawls a site width first. merlyn also adds links to spiders of his at this node (no less than four different ones!). Take your pick.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://153091]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2024-03-28 14:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found