Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Answer: How can I login to get data from a webpage?

by tinman (Curate)
on Apr 20, 2001 at 03:08 UTC ( #74025=categorized answer: print w/replies, xml ) Need Help??

Q&A > HTTP and FTP clients > How can I login to get data from a webpage? - Answer contributed by tinman

This is part of a script that I wrote for getting data from a password protected site. The site uses cookies for the logins, and the password URL can be passed straight in via the URL as GET parameters.

use LWP::UserAgent; use HTTP::Cookies; $ua = LWP::UserAgent->new; $ua->cookie_jar(HTTP::Cookies->new(file => "$cookie_file", autosave => 1)); $ua->timeout($url_timeout); $res = $ua->request(HTTP::Request->new('GET', "$login_url")); if ($res->is_error()) { $response = $response->status_line; exit; } $response = $ua->request(HTTP::Request->new('GET', "$data_url")); if ($response->is_error()) { $response = $response->status_line; } else { # do what you want to do with the pages here }

First I create a "cookie jar" which retains the cookie submitted by the site login mechanism.. Then, I submit the login_url, which is a complete URL escaped string. Thereafter, if the login is successful, I can navigate to any of the protected URLs..

Log In?

What's my password?
Create A New User
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2020-07-08 01:59 GMT
Find Nodes?
    Voting Booth?

    No recent polls found