Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re: search a foreign directory using LWP

by hdp (Beadle)
on Apr 25, 2001 at 08:44 UTC ( [id://75380] : note . print w/replies, xml ) Need Help??

in reply to search a foreign directory using LWP

Calling it a "directory" is a little misleading, because it's a HTML page, not a directory (in, for example, the opendir and readdir senses).

You'll need to parse the HTML and look for links to .txt files. I suggest HTML::Parser; you can try to do it with a regular expression, but it can get be incredibly hairy and lots of unneeded work (and I wouldn't suggest it).

Then you can use LWP to get each of the links (to text files) that you find and process them as you will.