Just another Perl shrine | |
PerlMonks |
Re: Re: Re: Cutting Out Previously Visited Web Pages in A Web Spiderby eric256 (Parson) |
on Mar 11, 2004 at 03:51 UTC ( [id://335697]=note: print w/replies, xml ) | Need Help?? |
If you are saving info on each page you find to a file then couldn't you just check to see if the file already exists before writing to it?? I didn't realy understand your code but you could save each url in a hash. Then just check to see if the url already exists in your hash before reading the page agian. The hash would only get as big the number of sites you spider. ___________ Eric Hodges
In Section
Seekers of Perl Wisdom
|
|