http://qs321.pair.com?node_id=1221955


in reply to help with writing a web crawler with LWP

Welcome to perl. Hope you enjoy your time with this language. Having experience with PHP, you will likely see a lot that looks familiar.

As to your question regarding a web crawler, may I suggest taking a look at Web Client Programming with Perl (available through the O'Reilly Open Books Project) and the LWP cookbook. It may seem a bit dated (1st. edition March 1997), but may provide a good background, and includes an example (in Chapter 6) of a recursive client. (You may also want to look at Merlyn's columns as well.)

Good luck with the project. Hope that helps.

  • Comment on Re: help with writing a web crawler with LWP