Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: need help determining which web browsing module to use

by marto (Cardinal)
on Nov 13, 2020 at 03:06 UTC ( [id://11123628]=note: print w/replies, xml ) Need Help??


in reply to need help determining which web browsing module to use

If you don't need JavaScript support, I'd use Mojo. Firstly an experiment to find the stuff:

#!/usr/bin/perl use strict; use warnings; use Mojo::DOM; use feature 'say'; my $html = '<div class="grouped-item product-purchase-wrapper-1">One< +/div><div class="grouped-item product-purchase-wrapper-7117">7117</d +iv>'; my $dom = Mojo::DOM->new( $html ); # find each div with a class beginning grouped-item product-purchase-w +rapper foreach my $div ( $dom->find('div[.class^=i"grouped-item product-purch +ase-wrapper"]')->each ){ say $div->text; }

Prints:

One 7117

Getting it from some live site:

#!/usr/bin/perl use strict; use warnings; use Mojo::UserAgent; use feature 'say'; my $ua = Mojo::UserAgent->new; my $url = 'https://urlgoeshere'; my $dom = $ua->get( $url )->res->dom; foreach my $div ( $dom->find('div[.class^=i"grouped-item product-purch +ase-wrapper"]')->each ){ say $div->text; }

See Mojo::DOM, Mojo::UserAgent, Super search for more mojo goodness.

Update: if you do need JavaScript support I'd suggest automating Chrome using WWW::Mechanize::Chrome, and using the xpath method.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11123628]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2024-04-18 14:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found