WWW::Mechanize and fooling server for javascript

gw1500se has asked for the wisdom of the Perl Monks concerning the following question:

This is a followup question to node ID 629420.

I have switched my method to use 'Mech' but now have a more basic problem. Since Mech does not support javascript I do not get the whole page or the form for submitting the login. I don't need javascript but I do need the source for the javascript (one of them contains a unique hash for encrypting the password). Maybe I am using the wring thing but when I use $mech->content(format=>'text'); I get an abbretiate form with the error that javascript is not enabled.

How do I fool this thing into thinking I have javascript enabled? This was not a problem when i was using LWP and 'GET'.

my $mech=WWW::Mechanize->new();
$mech->get($URL1);
print $mech->content(format=>'text');
[download]

Thanks.

Comment on WWW::Mechanize and fooling server for javascript Download Code

Replies are listed 'Best First'.
Re: WWW::Mechanize and fooling server for javascript by cLive ;-) (Prior) on Jul 31, 2007 at 00:04 UTC
Without seeing the HTML, I'll just throw out a best guess. My guess is that the page before uses javascript to set a form variable, so that when the form is submitted, if the dynamically generated form element is missing, it throws the Javascript error. That (or something like it) is the most likely problem. I suggest you install live HTTP headers plugin and watch what gets sent to the server when you manually go through the system in Firefox.	[reply]
Re^2: WWW::Mechanize and fooling server for javascript by spatterson (Pilgrim) on Jul 31, 2007 at 09:32 UTC
The "Javascript is not enabled" message could be generated by the server attempting browser-detection, so you may need to fake the useragent. just another cpan module author	[reply]
Re: WWW::Mechanize and fooling server for javascript by Cody Pendant (Prior) on Jul 31, 2007 at 01:21 UTC
Since Mech does not support javascript I do not get the whole page or the form for submitting the login. That doesn't make any sense at all. No, Mech doesn't support JavaScript but neither does LWP. Unsurprisingly since they're really the same thing. But of course Mech gets the full page, it couldn't possibly function otherwise. `print $mech->content()`, without the format, will show you the source. If you've got the source of the page, you can use it to find the Javascript. If you can find the JavaScript you can read it and find how it sets the data field. And if you can bear to tell us what site you're trying to log into, this will get a lot easier because we'll be able to look at it as well. Nobody says perl looks like line-noise any more kids today don't know what line-noise IS ...	[reply] [d/l]
Re^2: WWW::Mechanize and fooling server for javascript by gw1500se (Beadle) on Jul 31, 2007 at 12:04 UTC
I thought I replied to this last night but I don't see it. My apologies if I am missing something and this turns out to be a double post. Thanks for the reply. I thought 'content' returned an HTML object if I didn't use the format. I'll give that a try. I agree that it seems impossible to not get the whole page but since this is my first time with Mech I don't really know all it does yet. The bottom line of this exercise is to get ALL the javascript source including that which comes as a link rather then embedded. That was the problem I was having with LWP, the 'GET' only gave me the source for embedded javascript. For a better description of what I am doing, please see the explanation in Using LWP to automate a login. I am trying to extract the assigned IP address from the ISP for a DSL line. To do that I need to log on to a D-Link EBR-2310 which will serve a status page containing that information. The trick is to authenticate to the router.	[reply]
Re^3: WWW::Mechanize and fooling server for javascript by marto (Cardinal) on Jul 31, 2007 at 12:32 UTC
gw1500se, Some routers use JavaScript to dynamically create an html page using `document.write`. As you have already been told LWP and WWW::Mechanize do not support JavaScript, see how to reboot adsl modem with perl? for a similar problem, and list of WWW::Mechanize variants that do support JavaScript. I don't know what you mean by 'fooling Server' but depending on the JavaScript in question it is often possible to write some Perl that provides the same function as the JavaScript does. If all you are looking for is the IP address assigned by your ISP, perhaps you could simply use WWW::Mechanize to get http://www.whatismyip.com (or similar service), and parse the response. Martin	[reply] [d/l]
Re^4: WWW::Mechanize and fooling server for javascript by gw1500se (Beadle) on Jul 31, 2007 at 18:23 UTC
Re^5: WWW::Mechanize and fooling server for javascript by Corion (Patriarch) on Jul 31, 2007 at 18:29 UTC
Some notes below your chosen depth have not been shown here


No such thing as a small change
	PerlMonks