Need to pull a javascript generated webpage

miriamherald has asked for the wisdom of the Perl Monks concerning the following question:

I have a webpage that is generated almost entirely using javascript:

<div class="teamleaders">
<script LANGUAGE="JavaScript" SRC="http://www.mywebpage.com"></script>
[download]

Pulling this page just reveals all the javascript code rather than the html generated by the page. All I want to do is 'get' the file using an LWP agent and print the html to a new static file - I do not want the javascript code, I want rather the html that it generates. How can I do this?

Comment on Need to pull a javascript generated webpage Download Code

Replies are listed 'Best First'.
Re: Need to pull a javascript generated webpage by Corion (Patriarch) on Feb 03, 2010 at 17:28 UTC
Firefox can give you the HTML that results from Javascript being run in the browser. There is the Firefox View Rendered Source extension for that, and likely you can replicate that by automating Firefox with WWW::Mechanize::Firefox.	[reply]
Re^2: Need to pull a javascript generated webpage by miriamherald (Initiate) on Feb 03, 2010 at 17:45 UTC
Thanks Corion- how do you suggest I go about replicating it? I've used the Perl::Mechanize module before, but not Mechanize::Firefox. Can you give me a few more tips on how to implement this? I still feel lost.	[reply]
Re^3: Need to pull a javascript generated webpage by marto (Cardinal) on Feb 03, 2010 at 18:01 UTC
Did you even look at the WWW::Mechanize::Firefox documentation? "WWW::Mechanize::Firefox - use Firefox as if it were WWW::Mechanize" Use your WWW::Mechanize experience and apply it to this problem employing WWW::Mechanize::Firefox.	[reply]
Re: Need to pull a javascript generated webpage by pileofrogs (Priest) on Feb 03, 2010 at 19:04 UTC
You could also use lynx. http://lynx.isc.org http://home.pacific.net.sg/~kennethkwok/lynx/ Update: I suck!	[reply]
Re^2: Need to pull a javascript generated webpage by almut (Canon) on Feb 03, 2010 at 19:40 UTC
Hm, are you sure that lynx supports Javascript (to render the page)?	[reply]
Re^2: Need to pull a javascript generated webpage by redgreen (Priest) on Feb 03, 2010 at 23:48 UTC
Maybe you are thinking of ELinks, which does have some limited JavaScript support?	[reply]


laziness, impatience, and hubris
	PerlMonks