Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: LWP POST to a form on a 'secondary web page'

by andyford (Curate)
on Dec 25, 2006 at 19:02 UTC ( [id://591596]=note: print w/replies, xml ) Need Help??


in reply to LWP POST to a form on a 'secondary web page'

Can LWP only be used for shallow screen scraping of directly addressable web pages, and not for deeper navigation and extraction?
LWP should do the job. The only place I've gotten stuck using LWP was when the web server is using NTLM authentication.
LWP has some support for NTLM, but it seems to not work with newer implementations.

You will need to post some code to get good help though.

non-Perl: Andy Ford

  • Comment on Re: LWP POST to a form on a 'secondary web page'

Replies are listed 'Best First'.
Re^2: LWP POST to a form on a 'secondary web page'
by ady (Deacon) on Dec 25, 2006 at 20:35 UTC
    Tnx Andy,
    I got the NTLM authorization to work;
    I'm trying LWP partly out of curiosity and partly because I expect it to be faster than Win32::IE::Mechanize.
    I've updated the node with the relevant code.
    It runs as is, and does what I want it to do, but it's not as efficient as it could be, because the POST of the field values to the 'second page' doesn't work, and thus the database retrieves all values (ie doesn't filter by the fields in the form on the page)
    Best regards,
    allan
[reply]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://591596]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others sharing their wisdom with the Monastery: (1)
As of 2024-04-24 16:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found