Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
WWW::Mechanize::Chrome doesn't inherit from WWW::Mechanize, but it strives to provide the same API as WWW::Mechanize where possible/applicable.

Okay, and this because you don't have this in WMC:

use base qw(WWW::Mechanize);

right? I have to wonder if you considered other namespaces to put this, in particular when I found out that WWW::Mechanize inherits from LWP::UserAgent:

$ pwd /usr/local/share/perl/5.26.1/WWW/Mechanize $ .. $ ls Mechanize Mechanize.pm $ cat Mechanize.pm | more package WWW::Mechanize; #ABSTRACT: Handy web browsing in a Perl object use strict; use warnings; our $VERSION = '1.96'; use Tie::RefHash; use HTTP::Request 1.30; use LWP::UserAgent 5.827; use HTML::Form 1.00; use HTML::TokeParser; use Scalar::Util qw(tainted); use base 'LWP::UserAgent';
In some situations, I've deviated from the WWW::Mechanize API unfortunately, instead of either using a different method name or expanding the API in a compatible way...

From all of this background reading of what happened over the last 20 years of the internet and perl, the idea that one would want to faithfully represent in every detail what worked in 2003 with what works in 2015 seems like folly. I'll try out the new stuff and see how I do with it.

I hauled out one of my favorite WMG scripts only to find that it doesn't populate values correctly anymore, so I'm ready to start using newer tools. I have achieved such a minor amount of success. Between readmore tags I'll post the older script that I'm trying to modernize:

#! /usr/bin/perl use warnings; use strict; use 5.010; use WWW::Mechanize::GZip; use HTML::TableExtract qw(tree); use open ':std', OUT => ':utf8'; use Prompt::Timeout; use constant TIMEOUT => 3; use constant MAXTRIES => 30; ## redesign for solar eclipse of aug 21, 2017 my $site = 'http://www.fourmilab.ch/yoursky/cities.html'; my $mech = 'WWW::Mechanize::GZip'->new; $mech->get($site); $mech->follow_link( text => 'Portland OR' ); my $before_bound = 2457987.04167; #before conjunction my $after_bound = 2457988.0; #after conjunction $mech->set_fields(qw'date 2'); my $moon_seconds = 5; my $sun_seconds = 3; my $upper = $after_bound; my $lower = $before_bound; my $equal; my $equal_sec; my $now_string = localtime; my $filename = '2.pluto.txt'; open( my $jh, '>>', $filename ) or die "Could not open file '$filename +' $!"; say $jh "Script executed at $now_string"; say $jh " attempt upper lower guess"; my $attempts = 1; while ( ( $sun_seconds != $moon_seconds ) ) { my $default = ( ( $attempts >= MAXTRIES ) ) ? 'N' : 'Y'; my $answer = prompt( "Make query number $attempts?", $default, TIM +EOUT ); exit if $answer =~ /^N/i; my $guess = closetohalf( $upper, $lower ); say "to server is $attempts $upper $lower $guess "; say $jh "to server is $attempts $upper $lower $guess "; $mech->set_fields( jd => $guess ); $mech->click_button( value => "Update" ); my $te = 'HTML::TableExtract'->new; $te->parse( $mech->content ); my $table = ( $te->tables )[3]; my $table_tree = $table->tree; my $moon = $table_tree->cell( 5, 1 )->as_text; say "right ascension of moon $moon"; say $jh "right ascension of moon $moon"; my $sun = $table_tree->cell( 2, 1 )->as_text; say "right ascension of sun $sun"; say $jh "right ascension of sun $sun"; $moon_seconds = string_to_second($moon); say "moon seconds is $moon_seconds"; say $jh "moon seconds is $moon_seconds"; $sun_seconds = string_to_second($sun); say "sun seconds is $sun_seconds"; say $jh "sun seconds is $sun_seconds"; if ( $sun_seconds < $moon_seconds ) { $upper = $guess; } elsif ( $moon_seconds < $sun_seconds ) { $lower = $guess; } else { $equal = $guess; say "equal, while condition fails at julian day $equal"; say $jh "equal, while condition fails at julian day $equal"; $equal_sec = $moon_seconds; } $te->delete; $attempts++; } say $jh "equal seconds is $equal_sec"; # re-design 4-29-17 for solar eclipse sub string_to_second { my $string = shift; my $return = 9000; if ( my $success = $string =~ /^(\d*)h\s+(\d*)m\s+(\d*)s$/ ) { $return = 3600 * $1 + 60 * $2 + $3; } else { say "string was misformed"; } return $return; } sub closetohalf { my ( $up, $low ) = @_; $low + ( $up - $low ) * ( 0.4 + rand 0.2 ); }

I'm fairly confident that it behaved and produced accurate results. (There is a chance that it was a script that I was trying to extend and lost my way. I can't always tell them apart.) Now let's look at how far I've gotten with WMC:

#! /usr/bin/perl use warnings; use strict; use WWW::Mechanize::Chrome; use HTML::TableExtract qw(tree); use open ':std', OUT => ':utf8'; use Prompt::Timeout; use constant TIMEOUT => 3; use constant MAXTRIES => 30; ## redesign for solar eclipse of aug 21, 2017 ### begin 2020 rewrite ### with WWW::Mechanize::Chrome ### and Log::Log4perl use Log::Log4perl; use Data::Dump; use 5.016; my $log_conf3 = "/home/hogan/Documents/hogan/logs/conf_files/3.conf"; my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf"; #Log::Log4perl::init($log_conf3); #debug Log::Log4perl::init($log_conf4); #info my $logger = Log::Log4perl->get_logger(); my $current_level = $logger->level(); $logger->info("script begins with $current_level"); my $a = 'b'; for my $i ( 1 .. 2 ) { say "i is $i"; $logger->info("i is $i"); my $site = 'http://www.fourmilab.ch/yoursky/cities.html'; my $mech = WWW::Mechanize::Chrome->new( headless => 1, ); $mech->get($site); $mech->follow_link( text_regex => qr/Portland OR/i ); say "We are at " . $mech->uri; if ( $mech->success() ) { open my $gh, '>', "$a.form-log.txt" or warn "Couldn't open logfile $a.form-log.txt $!"; $mech->dump_forms($gh); say $gh "========="; } my $guess = 2458960; #Earth day 2020 in julian days $mech->form_number($i); say "$i works" if $mech->success(); say $mech->current_form->{name}; # ?? say "current form has a name" if $mech->success(); ## syntax that used to work with WWW::Mechanize # $mech->set_fields(qw'date 2'); #$mech->set_fields(); $mech->field( date => '2' ); ## analogs to set set_fields in WM say "first field set succeeded" if $mech->success(); $mech->field( jd => $guess ); say "second field set succeeded" if $mech->success(); $mech->click_button( value => "Update" ); # this seems similar to + WM say "clickbutton succeeded" if $mech->success(); my $string = $mech->uri; $logger->info("We are at $string") if $mech->success(); ## get a screenshot of how far we made it my $page_png = $mech->content_as_png(); my $base = '/home/hogan/5.scripts/1.corion./template_stuff/aimag +es'; my $fn = $base . "/$a.png"; open my $fh, '>', $fn or die "Couldn't create '$fn': $!"; binmode $fh, ':raw'; print $fh $page_png; close $fh; print "exiting show_screen with letter $a\n"; my $n = 2; $logger->info("sleeping for $n seconds ===================="); $mech->sleep($n); $a++; }

Terminal output:

$ ./5.pluto.pl script begins with 20000 i is 1 i is 1 Connected to ws://127.0.0.1:43601/devtools/browser/0c9af664-f568-4f8a- +bd7f-496dd9593030 We are at http://www.fourmilab.ch/cgi-bin/Yoursky?z=1&lat=45.5183&ns=N +orth&lon=122.676&ew=West 1 works Use of uninitialized value in say at ./5.pluto.pl line 55. current form has a name <no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line + 3779. <no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line + 3779. <no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line + 3779. 3 elements found for input with name 'date' at ./5.pluto.pl line 63. $

The log output shows that I've been jimmying with the loop values that I'm using to figure out which form I need. It turns out, it is not zero-based in this context. Zero bombs out.

2020/04/18 17:18:49 INFO script begins with 20000 2020/04/18 17:18:49 INFO Connected to ws://127.0.0.1:35721/devtools/br +owser/37b006e1-5a92-4191-b50f-7475ff4d12d9 2020/04/18 17:24:03 INFO script begins with 20000 2020/04/18 17:24:03 INFO i is 0 2020/04/18 17:24:03 INFO Connected to ws://127.0.0.1:38435/devtools/br +owser/79feea55-b183-4da4-9111-b43ddb825bdd 2020/04/18 17:26:33 INFO script begins with 20000 2020/04/18 17:26:33 INFO i is 1 2020/04/18 17:26:33 INFO Connected to ws://127.0.0.1:43601/devtools/br +owser/0c9af664-f568-4f8a-bd7f-496dd9593030

Whilst far short of a grand opus or masterpiece, this script does something that I couldn't manage with WM, namely, effective logging. Log::Log4Perl is required on WMC, which has the advantage of that functionality. The disadvantage is that you've gotta get it installed, which has been a (fixable) problem for some. It didn't want to install with my strawberry perl on windows 10, which I house on another partition. Two ways to solve the problem are given in getting Log::Log4perl to install on windows strawberry perl.

So, where am I stuck? Well, this is hot off the press and represents several similar attempts. It's nice to be using WMC and log4perl to figure this out. You can't be reading the same things the machines do as it overwhelms STDOUT. My partial results are encouraging, and this seems very much like a problem of getting forms and fields set and selected with new methods calls. Here is the uri we're looking at. It's a fun site, and you can readily enter your own information.

I do have data from the formdump:

[FORM] request /cgi-bin/Yoursky [INPUT (submit)] <no name> [INPUT (radio)] date [INPUT (radio)] date [INPUT (text)] utc [INPUT (radio)] date [INPUT (text)] jd [INPUT (text)] lat [INPUT (radio)] ns [INPUT (radio)] ns [INPUT (text)] lon [INPUT (radio)] ew [INPUT (radio)] ew [INPUT (checkbox)] coords [INPUT (checkbox)] moonp [INPUT (checkbox)] deep [INPUT (text)] deepm [INPUT (checkbox)] consto [INPUT (checkbox)] constn [INPUT (checkbox)] consta [INPUT (checkbox)] consts [INPUT (checkbox)] constb [INPUT (text)] limag [INPUT (checkbox)] starn [INPUT (text)] starnm [INPUT (checkbox)] starb [INPUT (text)] starbm [INPUT (checkbox)] flip [INPUT (text)] imgsize [INPUT (text)] fontscale [SELECT (select-one)] scheme [INPUT (checkbox)] edump [TEXTAREA (textarea)] elements

Fishing for tips. Thanks, Corion for your response and this considerable achievement:

$ wc -l $(locate Chrome.pm) 5761 /home/hogan/Documents/repos/wmc/WWW-Mechanize-Chrome/lib/WWW/Me +chanize/Chrome.pm 5708 /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm 11469 total $

In reply to Re^2: running an example script with WWW::Mechanize* module by Aldebaran
in thread running an example script with WWW::Mechanize* module by Aldebaran

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (2)
    As of 2021-01-19 02:50 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?
      Notices?