Anfield has asked for the wisdom of the Perl Monks concerning the following question:
There was an LWP spider script on perl website like 2 years ago. I can not find it any more now.
Does any one here have a similiar script? The script needed must be able to go deep at a pre-set value to fetch links.
thanx in advance
Re: LWP spider
by vagnerr (Prior) on Mar 20, 2002 at 19:58 UTC
|
When you install LWP a couple of perl scripts are installed at the same time called lwp-rget and lwp-request. lwp-rget is quite a nice spider program supporting basic auth and cookie files so that may be what you're looking for.
---If it doesn't fit use a bigger hammer | [reply] [Watch: Dir/Any] |
Re: LWP spider
by tachyon (Chancellor) on Mar 20, 2002 at 20:43 UTC
|
Link Checker is an LWP NET::FTP HTML::TokeParser link checking spider I wrote that crawls a site width first. merlyn also adds links to spiders of his at this node (no less than four different ones!). Take your pick.
cheers
tachyon
s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print
| [reply] [Watch: Dir/Any] |
|