Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^2: Retrieve Rendered Web Page Using WWW::Mechanize::Chrome

by roho (Bishop)
on Nov 03, 2020 at 14:28 UTC ( [id://11123369]=note: print w/replies, xml ) Need Help??


in reply to Re: Retrieve Rendered Web Page Using WWW::Mechanize::Chrome
in thread Retrieve Rendered Web Page Using WWW::Mechanize::Chrome

I get the same result using $mech->text(). I am using version 0.58 of WWW::Mechanize::Chrome.

"It's not how hard you work, it's how much you get done."

  • Comment on Re^2: Retrieve Rendered Web Page Using WWW::Mechanize::Chrome

Replies are listed 'Best First'.
Re^3: Retrieve Rendered Web Page Using WWW::Mechanize::Chrome
by Corion (Patriarch) on Nov 03, 2020 at 14:45 UTC

    Indeed this is weird. When I run your code, I get the Google consent modal, and an error

    2 elements found for //body at pb11123357.pl line 13.

    ... most likely because there are two body elements, one in the consent modal iframe and one in the main page.

    Once I click that away, I get the script content of the page, because the contents of script tags are also included in the textContent attribute. This is somewhat inconvenient, and I see no easy workaround for this. The fix seems to be to use the innerText attribute instead of textContent. Making this change makes the page "work" in the sense that the <script> content is not printed anymore. As a workaround you can monkey-patch the code until the next release:

    use WWW::Mechanize::Chrome; { no warnings 'redefine'; sub WWW::Mechanize::Chrome::text { my $self = shift; # Waugh - this is highly inefficient but conveniently short to wri +te # Maybe this should skip SCRIPT nodes... join '', map { $_->get_attribute('innerText') } $self->xpath('//bo +dy', single => 1 ); } }

    Thanks for reporting this! If this works for you as well, I'll write a test for this and release the fix soonish.

      I tried the workaround and it works fine. Thanks!

      "It's not how hard you work, it's how much you get done."

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11123369]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2024-04-19 05:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found