Re: Running JavaScript from within Perl

The JavaScript program on a web page can dynamically modify the page, so what you see has very little or no resemblance to the HTML source code! So, if you can scrape your web page using JavaScript, you get a peek at what's actually on the screen.

Here is an example. When you click on the "View HTML" button on this page, you'll see one thing. Then you click on the "Change" button which modifies the code, and then you click on View HTML again, and you'll see the code with some slight changes. The source code hasn't changed, but what's in the memory has changed, and when you get to harvest that, you get the real picture.

Here is the JavaScript program that harvests the HTML code:

var DATA = document.all[0].innerHTML;

If the block of HTML code you're trying to harvest is marked with an ID tag like this:

<DIV ID="Part3">
...

OR

<P ID="MyText">
...

OR

<TABLE ID="Table2">
...
[download]

then you don't need to harvest the entire HTML page. All you have to do is harvest whatever is tagged. So, you would just do this:

var DATA = document.getElementById("Part3").innerHTML;

Instead of using "innerHTML," you could also use "innerText" which gives you only the plain text without all the HTML tags and whatnot:

var DATA = document.getElementById("Part3").innerText;

Once you have the code in the DATA variable, then you can run a regex or something to get the actual number you're looking for.. JavaScript regex works like perl's.


<HTML>
<BODY>
<NOSCRIPT>
<DIV STYLE="BACKGROUND-COLOR:RED; COLOR:WHITE; FONT-FAMILY:ARIAL;"><CE
+NTER>This page requires JavaScript.</CENTER>
</DIV>
</NOSCRIPT>
<H3 ID="HEADING">Welcome</H3>
<DIV ID="CONTENT">
<P>This is a very simple HTML page.
<P><INPUT TYPE=BUTTON VALUE=" View HTML " onClick="ViewHTML();">
<INPUT TYPE=BUTTON VALUE=" Change " onClick="DoSomething();">
</DIV>

<SCRIPT>
function ViewHTML() {
  var DATA = document.all[0].innerHTML;
  alert("This is the page content as seen from JavaScript:\n\n" + DATA
+);
}

function DoSomething() {
  document.getElementById("HEADING").innerHTML = "<FONT COLOR=BLUE>DEA
+R VISITOR</FONT>";
  var MyCONTENT = document.getElementById("CONTENT");
  MyCONTENT.innerHTML = "<FONT COLOR=RED>" + MyCONTENT.innerHTML;
}
</SCRIPT>
[download]

I tested the above code, and it works in Firefox 52, KMeleon 7.5, QupZilla 1.8.6, Safari 5.1.7, Google Chrome 75, Internet Explorer 6, Opera 7.5, and Vivaldi 1.0. I have also tested it with an iPhone 7, Nokia Lumia 930 Windows Phone and an old Android 6 tablet. I haven't used any "ultra modern technology" that will break your phones. Everything in this example script is pretty standard.

Once you get the number you want to send back to your perl script, you could send it back by loading a picture:


<HTML>
<BODY>
<IMG NAME=PIX6 BORDER=0 WIDTH=1 HEIGHT=1 STYLE="POSITION:ABSOLUTE; TOP
+:0; LEFT:0;">

<SCRIPT>

NUMBER = 90;
document.images.PIX6.src = "http://www.yourwebsite.com/yourscript.pl?"
+ + NUMBER;
</SCRIPT>
[download]

Here you're sending the number 90 back to your perl script.

You could also signal to your perl script when somebody loads your web page with JavaScript turned off by putting a picture within the NOSCRIPT tags. Whatever you put between the NOSCRIPT tags will only appear when JavaScript is disabled on the page:

<NOSCRIPT>

<IMG SRC="http://www.yourwebsite.com/yourscript.pl?N" BORDER=0 WIDTH=1
+ HEIGHT=1 STYLE="POSITION:ABSOLUTE; TOP:0; LEFT:0;">

</NOSCRIPT>
[download]

Comment on Re: Running JavaScript from within Perl Select or Download Code

Replies are listed 'Best First'.
Re^2: Running JavaScript from within Perl by marto (Cardinal) on Sep 14, 2019 at 10:14 UTC
None of this addresses what OP is trying to achieve.	[reply]
Re^3: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 15, 2019 at 21:30 UTC
I'm having trouble understanding the WordPress.com REST API documentation. The example given for GET /sites/$site/posts/ is `curl 'https://public-api.wordpress.com/rest/v1.1/sites/en.blog.wordpre +ss.com/posts/?number=2'` [download] which I couldn't figure out how to make work. By contrast, the example provided in A Beginners’s Guide to the WordPress REST API is `curl -X GET -i http://the-art-of-autism.com/wp-json/wp/v2/posts` [download] which does work. Can you help me reconcile the two (which will hopefully help me interpret the rest of the WordPress REST API documentation)? Also, do REST API Resources only work on premium WordPress sites? I was able to execute GET /sites/$site/posts/ on the-art-of-autism.com (a premium site) but not on anautismobserver.wordpress.com (a free site). Do you know the reason for this? I really appreciate your help. You've already saved me a great deal of time and effort (and greatly increased my success chances). Thank you ever so much.	[reply] [d/l] [select]
Re^4: Running JavaScript from within Perl by marto (Cardinal) on Sep 16, 2019 at 08:48 UTC
"which I couldn't figure out how to make work." This works for me, I get back the JSON I'd expect given the endpoint. However this endpoint isn't the one I recommended. The remainder of this post is questions about WordPress setup/hosting, without looking at the associated documentation I don't know what the answers are.	[reply]
Re^3: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 18, 2019 at 19:15 UTC
In response to your short example using Mojo::UserAgent: (which I couldn't figure out how to respond to directly): I modified your code as follows to read url's from a file: use strict; use warnings; use Mojo::UserAgent; my $filename = 'urls_Mojo.txt'; open(my $fh, '<:encoding(UTF-8)', $filename) or die "Could not open file '$filename' $!"; my $y = 0; # input row count while (my $row = <$fh>) { $y++; print $y; print " $row"; my $url = $row; # create a Mojo:UserAgent my $ua = Mojo::UserAgent->new; # use $ua to get the url and assign the value of 'subscriber_count' in + the json # to avariable, $subscribers my $subscribers = $ua->get( $url )->result->json->{subscribers_count}; # print the variable to screen print "Number of subscribers: $subscribers\n"; } [download] it worked when the file 'urls_Mojo.txt' contained `https://public-api.wordpress.com/rest/v1/read/feed/http%3A%2F%2Fthe-ar +t-of-autism.com%2Ffeed` [download] but gave a "Can't use an undefined value as a HASH reference" error when I added a second line to 'urls_Mojo.txt' as follows: `https://public-api.wordpress.com/rest/v1/read/feed/http%3A%2F%2Fthe-ar +t-of-autism.com%2Ffeed https://public-api.wordpress.com/rest/v1/read/feed/http%3A%2F%2Fanauti +smobserver.wordpress.com%2Ffeed` [download] Can you help me figure out how to apply this script to a list of url's in a file? Thanks.	[reply] [d/l] [select]
Re^4: Running JavaScript from within Perl by hippo (Bishop) on Sep 18, 2019 at 22:08 UTC
You have not chomped `$row` (as can be seen from the printed output when run). If I do that it works fine. `$ diff 11106348_orig.pl 11106348.pl 12a13 > chomp $row;` [download] I presume when you used just one URL in the data file you did not even add the EOL character so it worked as a fluke.	[reply] [d/l] [select]
Re^4: Running JavaScript from within Perl by marto (Cardinal) on Sep 19, 2019 at 08:53 UTC
As hippo mentioned, chomp. See also https://perlmaven.com/open-and-read-from-files: "In this article we see how to do this with core perl, but there are more modern and nicer ways to do this using Path::Tiny to read files." In addition to that you should consider properly indenting your code to make it easier to follow.	[reply]
Re^3: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 17, 2019 at 05:54 UTC
Using GET /read/feed/$feed_url_or_id I can generate a web page containing the number of followers shown as "subscribers_count". How do I read this page into a perl script? I tried HTML::TreeBuilder and got the error message: https://public-api.wordpress.com/rest/v1/read/feed/http%3A%2F%2Fthe-art-of-autism.com%2Ffeed returned application/json not HTML Should I use WWW::Mechanize::Chrome, JSON, JavaScript, or something else? How do I provide them input from a URL?	[reply]
Re^4: Running JavaScript from within Perl by marto (Cardinal) on Sep 17, 2019 at 06:24 UTC
Here's a short example using Mojo::UserAgent: `#!/usr/bin/perl use strict; use warnings; use Mojo::UserAgent; # your URL my $url = 'https://public-api.wordpress.com/rest/v1/read/feed/http%3A%2F%2Fthe-a +rt-of-autism.com%2Ffeed'; # create a Mojo:UserAgent my $ua = Mojo::UserAgent->new; # use $ua to get the url and assign the value of 'subscriber_count' in + the json # to avariable, $subscribers my $subscribers = $ua->get( $url )->result->json->{subscribers_count}; # print the variable to screen print "Number of subscribers: $subscribers\n";` [download] Running this returns: `Number of subscribers: 181` [download] To install the required module, from the command line: `cpanm Mojolicious (or if you don't have cpanm) cpan Mojolicious` [download]	[reply] [d/l] [select]
Re^4: Running JavaScript from within Perl by bliako (Monsignor) on Sep 17, 2019 at 08:50 UTC
To read json data from file, optionally utf8-encoded, into a Perl data structure I do: use strict; use warnings; # use this to enable script working with utf8 strings use utf8; # use this in order to print utf8 to console binmode STDOUT, ":utf8"; binmode STDERR, ":utf8"; # contains json-encode/decode routines use JSON; # contains utf8 encode/decode routines use Encode; # where json data is: my $jsonfile = 'data.json'; # open file with json my $INP; die "opening '$jsonfile'" unless open $INP, '<:encoding(UTF-8)', $json +file; # slurp file contents my $contents; { local $/ = undef; $contents = Encode::encode 'UTF-8', <$INP> } close + $INP; die "nothing to convert in '$jsonfile'..." unless $contents; # convert json text to perl data structure my $perldata = JSON::decode_json $contents; die "JSON::decode_json() has failed" unless $perldata; # dump the data for debugging use Data::Dumper; # only needed if you want to dump a data structure print Dumper($perldata); # access element of data print "image is ".$perldata->{'image'}."\n"; print "description (has unicode) is ".$perldata->{'description'}."\n"; print "meta/links/self is ".$perldata->{'meta'}->{'links'}->{'self'}." +\n"; [download] bw, bliako	[reply] [d/l]
Re^5: Running JavaScript from within Perl by Anonymous Monk on Sep 17, 2019 at 09:16 UTC
Re^6: Running JavaScript from within Perl by bliako (Monsignor) on Sep 17, 2019 at 11:01 UTC
Re^3: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 18, 2019 at 18:24 UTC
When I type `curl --help` [download] I get a list of options. Does that mean I have curl installed? (I don't remember installing it.) If not, please tell me how to install it from the zip file. Thanks.	[reply] [d/l]
Re^4: Running JavaScript from within Perl by marto (Cardinal) on Sep 18, 2019 at 18:43 UTC
Sounds like you have curl alright. Re^4: Running JavaScript from within Perl shoes a perl solution, like you asked for ;)	[reply]
Re^3: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 19, 2019 at 17:45 UTC
Do you know of a way to indent a script automatically? I currently use Notepad++ and also have Komodo IDE installed (but would happily use another editor that can indent automatically).	[reply]
Re^4: Running JavaScript from within Perl (indentation) by hippo (Bishop) on Sep 19, 2019 at 18:08 UTC
perltidy is an IDE/Editor-agnostic way to indent your code in a controlled manner. Any IDE/Editor should be able to hook into it at least on write-to-disk.	[reply]
Re^4: Running JavaScript from within Perl by marto (Cardinal) on Sep 19, 2019 at 17:55 UTC
I didn't, but a web search found the answer, un surprisingly :) https://stackoverflow.com/questions/412427/auto-indent-in-notepad	[reply]
Re^4: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 20, 2019 at 17:51 UTC
When I typed "perltidy" I got the following response: `Can't locate Perl/Tidy.pm in @INC (you may need to install the Perl::T +idy module)` [download] When I typed "cpanm Perl::Tidy" in the Command Window the installation failed, with the following in the build.log: cpanm (App::cpanminus) 1.7044 on perl 5.028001 built for MSWin32-x64-m +ulti-thread Work directory is C:\Users\Judy/.cpanm/work/1569000722.23204 You have make C:\Strawberry\c\bin\dmake.exe You have C:\WINDOWS\system32\curl.exe Falling back to Archive::Tar 2.30 Searching Perl::Tidy () on cpanmetadb ... --> Working on Perl::Tidy Fetching http://www.cpan.org/authors/id/S/SH/SHANCOCK/Perl-Tidy-201909 +15.tar.gz -> OK Unpacking Perl-Tidy-20190915.tar.gz Entering Perl-Tidy-20190915 Checking configure dependencies from META.json Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.34) Configuring Perl-Tidy-20190915 Running Makefile.PL Checking if your kit is complete... Looks good Generating a dmake-style Makefile Writing Makefile for Perl::Tidy Writing MYMETA.yml and MYMETA.json -> OK Checking dependencies from MYMETA.json ... Checking if you have ExtUtils::MakeMaker 0 ... Yes (7.34) Building and testing Perl-Tidy-20190915 ############################################################# ### ### ### DMAKE WARNING ### ### ### ### Do not use dmake.exe utility as it is no longer ### ### part of Strawberry Perl, use gmake.exe instead! ### ### ### ### If you have troubles with CPAN client delete: ### ### %USERPROFILE%\AppData\Local\.cpan\CPAN\MyConfig.pm ### ### ### ############################################################# -> FAIL Installing Perl::Tidy failed. See C:\Users\Judy\.cpanm\work\15 +69000722.23204\build.log for details. Retry with --force to force ins +tall it. Expiring 5 work directories. [download] Please tell me how to install perltidy. Thanks.	[reply] [d/l] [select]
Re^5: Running JavaScript from within Perl by Corion (Patriarch) on Sep 20, 2019 at 17:58 UTC
Re^4: Running JavaScript from within Perl by Anonymous Monk on Sep 19, 2019 at 19:52 UTC
Do you know of a way to indent a script automatically? I currently use Notepad++ and also have Komodo IDE installed (but would happily use another editor that can indent automatically). Komodo has brilliant auto-indent features. Open preferences and select "indentation" to check out the global and per language indentation settings like (quoted from komodo docs): "Smart Indent: Komodo automatically anticipates logical indentation points, based on language cues (such as open braces)." "Auto-adjust closing braces: Komodo automatically aligns closing braces with corresponding opening braces." "Show indentation guides: Select to display indentation markers (grey vertical lines). An indentation marker is displayed every time the number of spaces on the left margin equals the value specified in the Number of spaces per indent field." This works great with prefs > smart editing > enable soft characters ("Enable soft characters turns on language-specific autocompletion for brackets, braces and other delimiters. The highlighted closing character is inserted after the cursor and can be “typed over”.")	[reply]
Re^2: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 15, 2019 at 08:31 UTC
I don't mind responses that go beyond the narrow bounds of what I asked. It's like learning a new language: sometimes it's best to immerse myself in the new culture and see what I can absorb. I like learning new software through following tutorials (though this runs the risk of learning outdated information). I'm starting working through The Ultimate Guide To The WordPress REST API (written in September 2015 by Josh Pollock). He recommends using Vagrant, VirtualBox, and Git, which I've downloaded and installed on my computer Is The Ultimate Guide To The WordPress REST API a good resource (obtained from here)? Do you know of any better (perhaps newer) tutorials for the WordPress REST API?	[reply]
Re^3: Running JavaScript from within Perl by marto (Cardinal) on Sep 15, 2019 at 09:26 UTC
"I don't mind responses that go beyond the narrow bounds of what I asked. It's like learning a new language: sometimes it's best to immerse myself in the new culture and see what I can absorb." The method described in the post you're replying to wont help you achieve what you asked to do. If you're interested in learning about JavaScript and HTML/DOM manipulation there are better resources (from the Mojolicious docs): "All web development starts with HTML, CSS and JavaScript, to learn the basics we recommend the Mozilla Developer Network. And if you want to know more about how browsers and web servers actually communicate, there's also a very nice introduction to HTTP." "I've downloaded and installed Vagrant, VirtualBox, and Git for Windows on my computer" What part of problem does this solve? "Is this a good resource? (https://wpengine.com/resources/the-ultimate-guide-to-the-wordpress-rest-api/)" I've no idea, you need to register to download an ebook. "Do you know of any better (perhaps newer) tutorials for the WordPress REST API?" What is missing from the official WordPress documentation? Update: Re^3: Running JavaScript from within Perl (or just use the API)/https://developer.wordpress.com/docs/api/1.1/get/sites/%24site/stats/followers/.	[reply]
Re^4: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 15, 2019 at 17:09 UTC
Here's a link to the book: http://cdn2.hubspot.net/hubfs/298401/Documents_for_download/WP-API-ebook_final_09162015.pdf It offers a step-by-step guide to using the WordPress REST API. It recommends using VVV (Vagrant), and somewhere in the instructions were also recommendations for VirtualBox and Git. What's missing from the official WordPress documentation is a step-by-step guide to using the WordPress REST API for neophytes like me.	[reply]
Re^5: Running JavaScript from within Perl by marto (Cardinal) on Sep 15, 2019 at 18:32 UTC
Re^6: Running JavaScript from within Perl by anautismobserver (Sexton) on Sep 15, 2019 at 18:48 UTC
Some notes below your chosen depth have not been shown here


go ahead... be a heretic
	PerlMonks