Problems? Is your data what you think it is? | |
PerlMonks |
Re: Retrieve a CGI pageby dani++ (Sexton) |
on Aug 31, 2001 at 14:58 UTC ( [id://109349]=note: print w/replies, xml ) | Need Help?? |
I've written a fairly sophisticated html spider using perl, lynx and tcsh (as glue), all the pages accessed are CGIs and it works as advertised. Have you tried to use 'lynx --source' or 'lynx --dump' as suggested?
I've refrained from using LWP as the target CGI system required cookies, sessions and full browser support. Moreover, Lynx has a limited script option '-cmd_script=<script file>' that you can use to program what it does (download files, etc). Use '-cmd_log=<script log file>' to learn the syntax of the script files. In my case I first download the pages, use perl to parse and analyse them, build a custom lynx script to retrieve exactly the data I want and run lynx again with the generated script.
In Section
Seekers of Perl Wisdom
|
|