sailortailorson has asked for the wisdom of the Perl Monks concerning the following question:

O Denizens of the Cloister of the Fruit of the Oyster,

Using LWP::UserAgent, when I look at the content of a page that has lots of Ajaxy stuff, I see the Javascript, but not the results of the Javascript.

In some cases, I need to see the final page, after all the Javascript has altered the page. Is there a way to get this from the U.A.?

For the present, I have switched over to using IE as driven by Win32::OLE, but if it is possible to do in LWP, or some other more Perlish way, I would rather do that.

Also (in a somewhat related question) I have searched CPAN for 'Greasemonkey' related scripts, but find nothing. Is anyone doing anything that simulates Greasemonkey for LWP scripts?

Just another Nabob of Nacre

Replies are listed 'Best First'.
Re: LWP and Javascript
by Joost (Canon) on Mar 29, 2007 at 22:28 UTC

      Thanks. This all looks like the kind of things I need to add to a general toolbox for making webpages jump through hoops, mostly for automating testing, and computer aided manual testing. The specific task I wanted to solve is counting sponsored links (ads) on a web page. I cannot see them in LWP, only the Ajax that goes and gets them.

      For some reason, I have not even tried the mechanize modules. I guess I thought I was getting more control from LWP. I just heard about Selenium last week for the first time, so I will take a look at that too.

      In regard to your note about LWP, that confirms the sense I had about it. So I know now not to bark up that tree anymore.

        WWW::Mechanize is a subclass of LWP::UserAgent, so more or less anything you can do with LWP::UA you can do with WWW::Mechanize. Mechanize is just easier to use when automating click-trails (start at link such-and-such, click link named "bla bla", fill in form field "thingy", submit - etc)

        The *::Mechanize::* modules linked in my post above mimic and extend WWW::Mechanize's behaviour but are probably not 100% API compatible - YMMV.