Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Way to grab webpages WITH cookies enabled?

by Kratimoyporroon (Novice)
on Feb 15, 2002 at 06:26 UTC ( [id://145632]=perlquestion: print w/replies, xml ) Need Help??

Kratimoyporroon has asked for the wisdom of the Perl Monks concerning the following question:

Hi. I am relatively new to Perl and am writing a script that will download a certain webpage for me and search for some things that I desire. However to download this webpage properly, I need to send it my cookies (or else I won't get the correct information). Currently I am using a module named LWP::Simple that has a get function that will download a webpage for me. Unfortunately it does not use my cookies. Are there any modules that will do what I want? Just to reiterate quickly, I want to download a webpage programmatically, and to get the correct webpage when I download from the URL, I need to be able to send it my cookies. Is there a module? Thanks a lot.
  • Comment on Way to grab webpages WITH cookies enabled?

Replies are listed 'Best First'.
Re: Way to grab webpages WITH cookies enabled?
by Chmrr (Vicar) on Feb 15, 2002 at 06:41 UTC

    As discussed in the chatterbox, I think that this answers your question best, of the multiple times it has been answered.

    perl -pe '"I lo*`+$^X$\"$]!$/"=~m%(.*)%s;$_=$1;y^`+*^e v^#$&V"+@( NO CARRIER'

      Hmm. I've got the user-agent able to download the webpage ok, but I'm still having trouble with the cookies. Apparently HTTP::Cookies uses its own cookie file. However I have the cookie on my machine in internet explorer format. There's no way for me (that _I_ know of at least :)) to get the cookie using the user agent, because that requires that I submit a form. (i.e. I submit a form with login + pass, it logs me in and sets the cookie). Now I have already logged in and told it to store my cookie - how do I "transfer" this cookie to HTTP::Cookies?

        See lwpcook, specifically the section on "Cookies." It is indeed possible to put together a request which "submits" a form, gets the cookie that is returned, and use it in later requests. This is an example of such, in fact, even using PerlMonks as an example.

        Alternatively, you can find the file on your hard drive where IE stores the cookie, look at the name and value, and hard-code those in as a cookie.

        On the gripping hand, you can visit the site in Netscape or compatible browser, and get a cookie. HTTP::Cookies knows how to read Netscape cookie files, and you can just tell it to read the cookie file that Netscape has produced with the cookie you want.

        You'd almost get the feeling that There Is More Than One Way To Do It, wouldn't you? :)

        perl -pe '"I lo*`+$^X$\"$]!$/"=~m%(.*)%s;$_=$1;y^`+*^e v^#$&V"+@( NO CARRIER'

Re: Way to grab webpages WITH cookies enabled?
by dhable (Monk) on Feb 15, 2002 at 14:53 UTC
    If you really wanted a challenge and were running on Windows, you could always use PerlCOM to manipulate the IE browser enginer using IWebBrowser. It's bloated and slow, and like all M$ stuff it would leak the cookies to the site if it requested it.

    I feel so dirty....must take shower.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://145632]
Approved by root
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2024-04-21 07:53 GMT
Find Nodes?
    Voting Booth?

    No recent polls found