Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Scraping Javascript page using perl

by bliako (Monsignor)
on Nov 21, 2020 at 17:23 UTC ( [id://11123983]=note: print w/replies, xml ) Need Help??


in reply to Scraping Javascript page using perl

Quick fix: Open Firefox's Developer Tools, go to the Network tab and observe all the transactions which happen during loading. One of them is requesting the HTML for the actual contents of that page, the 1892 theses (alas only mathematics is immortal), something like this:

http://operedigitali.lincei.it/rendicontiFMN/rol/visart.php?lang=it&type=mat&serie=5&anno=1892&volume=1

The longer way which is the "proper" way IMO is to do what Corion suggested and use WWW::Mechanize::Chrome (I am not acquainted with WWW::Mechanize::PhantomJS). This instructs google chrome browser to get the web-page and then asks it to provide Perl with the DOM. Then it's straight forward with XPath selectors to reach and suck out the desired div's contents.

bw, bliako

Replies are listed 'Best First'.
Re^2: Scraping Javascript page using perl
by Bpl (Scribe) on Nov 21, 2020 at 19:12 UTC
    Hi Bliako, nice to see you again, Many thanks for the help!

    The project is growing, you'll have more info next days.

    Still thanks for everything!

      Glad you never give up! All the best

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11123983]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2024-04-25 18:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found