Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

LWP and Javascript

by sailortailorson (Scribe)
on Mar 29, 2007 at 22:19 UTC ( #607370=perlquestion: print w/replies, xml ) Need Help??

sailortailorson has asked for the wisdom of the Perl Monks concerning the following question:

O Denizens of the Cloister of the Fruit of the Oyster,

Using LWP::UserAgent, when I look at the content of a page that has lots of Ajaxy stuff, I see the Javascript, but not the results of the Javascript.

In some cases, I need to see the final page, after all the Javascript has altered the page. Is there a way to get this from the U.A.?

For the present, I have switched over to using IE as driven by Win32::OLE, but if it is possible to do in LWP, or some other more Perlish way, I would rather do that.

Also (in a somewhat related question) I have searched CPAN for 'Greasemonkey' related scripts, but find nothing. Is anyone doing anything that simulates Greasemonkey for LWP scripts?

Just another Nabob of Nacre

Replies are listed 'Best First'.
Re: LWP and Javascript
by Joost (Canon) on Mar 29, 2007 at 22:28 UTC
      Joost,

      Thanks. This all looks like the kind of things I need to add to a general toolbox for making webpages jump through hoops, mostly for automating testing, and computer aided manual testing. The specific task I wanted to solve is counting sponsored links (ads) on a web page. I cannot see them in LWP, only the Ajax that goes and gets them.

      For some reason, I have not even tried the mechanize modules. I guess I thought I was getting more control from LWP. I just heard about Selenium last week for the first time, so I will take a look at that too.

      In regard to your note about LWP, that confirms the sense I had about it. So I know now not to bark up that tree anymore.

      Thanks
        WWW::Mechanize is a subclass of LWP::UserAgent, so more or less anything you can do with LWP::UA you can do with WWW::Mechanize. Mechanize is just easier to use when automating click-trails (start at link such-and-such, click link named "bla bla", fill in form field "thingy", submit - etc)

        The *::Mechanize::* modules linked in my post above mimic and extend WWW::Mechanize's behaviour but are probably not 100% API compatible - YMMV.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://607370]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (7)
As of 2022-08-11 17:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?