http://qs321.pair.com?node_id=11115764


in reply to Re: running an example script with WWW::Mechanize* module
in thread running an example script with WWW::Mechanize* module

WWW::Mechanize::Chrome doesn't inherit from WWW::Mechanize, but it strives to provide the same API as WWW::Mechanize where possible/applicable.

Okay, and this because you don't have this in WMC:

use base qw(WWW::Mechanize);

right? I have to wonder if you considered other namespaces to put this, in particular when I found out that WWW::Mechanize inherits from LWP::UserAgent:

$ pwd
/usr/local/share/perl/5.26.1/WWW/Mechanize
$ ..
$ ls
Mechanize  Mechanize.pm
$ cat Mechanize.pm | more
package WWW::Mechanize;

#ABSTRACT: Handy web browsing in a Perl object


use strict;
use warnings;

our $VERSION = '1.96';

use Tie::RefHash;
use HTTP::Request 1.30;
use LWP::UserAgent 5.827;
use HTML::Form 1.00;
use HTML::TokeParser;
use Scalar::Util qw(tainted);

use base 'LWP::UserAgent';

[download]
In some situations, I've deviated from the WWW::Mechanize API unfortunately, instead of either using a different method name or expanding the API in a compatible way...

From all of this background reading of what happened over the last 20 years of the internet and perl, the idea that one would want to faithfully represent in every detail what worked in 2003 with what works in 2015 seems like folly. I'll try out the new stuff and see how I do with it.

I hauled out one of my favorite WMG scripts only to find that it doesn't populate values correctly anymore, so I'm ready to start using newer tools. I have achieved such a minor amount of success. Between readmore tags I'll post the older script that I'm trying to modernize:

#! /usr/bin/perl
use warnings;
use strict;
use 5.010;
use WWW::Mechanize::GZip;
use HTML::TableExtract qw(tree);
use open ':std', OUT => ':utf8';
use Prompt::Timeout;
use constant TIMEOUT  => 3;
use constant MAXTRIES => 30;

## redesign for solar eclipse of aug 21, 2017

my $site = 'http://www.fourmilab.ch/yoursky/cities.html';
my $mech = 'WWW::Mechanize::GZip'->new;
$mech->get($site);
$mech->follow_link( text => 'Portland OR' );
my $before_bound = 2457987.04167;    #before conjunction
my $after_bound  = 2457988.0;        #after conjunction
$mech->set_fields(qw'date 2');
my $moon_seconds = 5;
my $sun_seconds  = 3;
my $upper        = $after_bound;
my $lower        = $before_bound;
my $equal;
my $equal_sec;
my $now_string = localtime;
my $filename   = '2.pluto.txt';
open( my $jh, '>>', $filename ) or die "Could not open file '$filename
+' $!";
say $jh "Script executed at $now_string";
say $jh "       attempt  upper  lower  guess";
my $attempts = 1;

while ( ( $sun_seconds != $moon_seconds ) ) {

    my $default = ( ( $attempts >= MAXTRIES ) ) ? 'N' : 'Y';
    my $answer = prompt( "Make query number $attempts?", $default, TIM
+EOUT );
    exit if $answer =~ /^N/i;

    my $guess = closetohalf( $upper, $lower );
    say "to server is $attempts $upper $lower $guess ";
    say $jh "to server is $attempts $upper $lower $guess ";
    $mech->set_fields( jd => $guess );
    $mech->click_button( value => "Update" );
    my $te = 'HTML::TableExtract'->new;
    $te->parse( $mech->content );
    my $table      = ( $te->tables )[3];
    my $table_tree = $table->tree;
    my $moon       = $table_tree->cell( 5, 1 )->as_text;
    say "right ascension of moon $moon";
    say $jh "right ascension of moon $moon";
    my $sun = $table_tree->cell( 2, 1 )->as_text;
    say "right ascension of sun $sun";
    say $jh "right ascension of sun $sun";
    $moon_seconds = string_to_second($moon);
    say "moon seconds is $moon_seconds";
    say $jh "moon seconds is $moon_seconds";
    $sun_seconds = string_to_second($sun);
    say "sun seconds is $sun_seconds";
    say $jh "sun seconds is $sun_seconds";

    if ( $sun_seconds < $moon_seconds ) {
        $upper = $guess;
    }
    elsif ( $moon_seconds < $sun_seconds ) {
        $lower = $guess;
    }
    else {
        $equal = $guess;
        say "equal, while condition fails at julian day $equal";
        say $jh "equal, while condition fails at julian day $equal";
        $equal_sec = $moon_seconds;
    }
    $te->delete;
    $attempts++;
}

say $jh "equal seconds is $equal_sec";

# re-design 4-29-17 for solar eclipse

sub string_to_second {
    my $string = shift;
    my $return = 9000;
    if ( my $success = $string =~ /^(\d*)h\s+(\d*)m\s+(\d*)s$/ ) {
        $return = 3600 * $1 + 60 * $2 + $3;
    }
    else {
        say "string was misformed";
    }
    return $return;
}

sub closetohalf {
    my ( $up, $low ) = @_;
    $low + ( $up - $low ) * ( 0.4 + rand 0.2 );
}

[download]

I'm fairly confident that it behaved and produced accurate results. (There is a chance that it was a script that I was trying to extend and lost my way. I can't always tell them apart.) Now let's look at how far I've gotten with WMC:

#! /usr/bin/perl
use warnings;
use strict;
use WWW::Mechanize::Chrome;
use HTML::TableExtract qw(tree);
use open ':std', OUT => ':utf8';
use Prompt::Timeout;
use constant TIMEOUT  => 3;
use constant MAXTRIES => 30;

## redesign for solar eclipse of aug 21, 2017

### begin 2020 rewrite
### with WWW::Mechanize::Chrome
### and Log::Log4perl

use Log::Log4perl;
use Data::Dump;
use 5.016;

my $log_conf3 = "/home/hogan/Documents/hogan/logs/conf_files/3.conf";
my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf";

#Log::Log4perl::init($log_conf3); #debug
Log::Log4perl::init($log_conf4);    #info
my $logger        = Log::Log4perl->get_logger();
my $current_level = $logger->level();
$logger->info("script begins with $current_level");
my $a = 'b';

for my $i ( 1 .. 2 ) {

  say "i is $i";
  $logger->info("i is $i");
  my $site = 'http://www.fourmilab.ch/yoursky/cities.html';
  my $mech = WWW::Mechanize::Chrome->new( headless => 1, );
  $mech->get($site);

  $mech->follow_link( text_regex => qr/Portland OR/i );
  say "We are at " . $mech->uri;

  if ( $mech->success() ) {
    open my $gh, '>', "$a.form-log.txt"
      or warn "Couldn't open logfile $a.form-log.txt $!";
    $mech->dump_forms($gh);
    say $gh "=========";
  }

  my $guess = 2458960;    #Earth day 2020 in julian days

  $mech->form_number($i);
  say "$i works"
    if $mech->success();
  say $mech->current_form->{name};    # ??
  say "current form has a name"
    if $mech->success();

## syntax that used to work with WWW::Mechanize
  #   $mech->set_fields(qw'date 2');

  #$mech->set_fields();
  $mech->field( date => '2' );    ## analogs to set set_fields in WM
  say "first field set succeeded"
    if $mech->success();
  $mech->field( jd => $guess );
  say "second field set succeeded"
    if $mech->success();
  $mech->click_button( value => "Update" );    # this seems similar to
+ WM
  say "clickbutton succeeded"
    if $mech->success();
  my $string = $mech->uri;
  $logger->info("We are at $string")
    if $mech->success();

## get a screenshot of how far we made it
  my $page_png = $mech->content_as_png();
  my $base     = '/home/hogan/5.scripts/1.corion./template_stuff/aimag
+es';
  my $fn       = $base . "/$a.png";
  open my $fh, '>', $fn
    or die "Couldn't create '$fn': $!";
  binmode $fh, ':raw';
  print $fh $page_png;
  close $fh;
  print "exiting show_screen with letter $a\n";
  my $n = 2;
  $logger->info("sleeping for $n seconds  ====================");
  $mech->sleep($n);
  $a++;
}

[download]

Terminal output:

$ ./5.pluto.pl 
script begins with 20000 
i is 1
i is 1 
Connected to ws://127.0.0.1:43601/devtools/browser/0c9af664-f568-4f8a-
+bd7f-496dd9593030 
We are at http://www.fourmilab.ch/cgi-bin/Yoursky?z=1&lat=45.5183&ns=N
+orth&lon=122.676&ew=West
1 works
Use of uninitialized value in say at ./5.pluto.pl line 55.

current form has a name
<no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line
+ 3779.
<no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line
+ 3779.
<no text> at /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm line
+ 3779.
3 elements found for input with name 'date' at ./5.pluto.pl line 63.
$

[download]

The log output shows that I've been jimmying with the loop values that I'm using to figure out which form I need. It turns out, it is not zero-based in this context. Zero bombs out.

2020/04/18 17:18:49 INFO script begins with 20000 
2020/04/18 17:18:49 INFO Connected to ws://127.0.0.1:35721/devtools/br
+owser/37b006e1-5a92-4191-b50f-7475ff4d12d9 
2020/04/18 17:24:03 INFO script begins with 20000 
2020/04/18 17:24:03 INFO i is 0 
2020/04/18 17:24:03 INFO Connected to ws://127.0.0.1:38435/devtools/br
+owser/79feea55-b183-4da4-9111-b43ddb825bdd 
2020/04/18 17:26:33 INFO script begins with 20000 
2020/04/18 17:26:33 INFO i is 1 
2020/04/18 17:26:33 INFO Connected to ws://127.0.0.1:43601/devtools/br
+owser/0c9af664-f568-4f8a-bd7f-496dd9593030

[download]

Whilst far short of a grand opus or masterpiece, this script does something that I couldn't manage with WM, namely, effective logging. Log::Log4Perl is required on WMC, which has the advantage of that functionality. The disadvantage is that you've gotta get it installed, which has been a (fixable) problem for some. It didn't want to install with my strawberry perl on windows 10, which I house on another partition. Two ways to solve the problem are given in getting Log::Log4perl to install on windows strawberry perl.

So, where am I stuck? Well, this is hot off the press and represents several similar attempts. It's nice to be using WMC and log4perl to figure this out. You can't be reading the same things the machines do as it overwhelms STDOUT. My partial results are encouraging, and this seems very much like a problem of getting forms and fields set and selected with new methods calls. Here is the uri we're looking at. It's a fun site, and you can readily enter your own information.

I do have data from the formdump:

[FORM] request /cgi-bin/Yoursky
    [INPUT (submit)] <no name>
    [INPUT (radio)] date
    [INPUT (radio)] date
    [INPUT (text)] utc
    [INPUT (radio)] date
    [INPUT (text)] jd
    [INPUT (text)] lat
    [INPUT (radio)] ns
    [INPUT (radio)] ns
    [INPUT (text)] lon
    [INPUT (radio)] ew
    [INPUT (radio)] ew
    [INPUT (checkbox)] coords
    [INPUT (checkbox)] moonp
    [INPUT (checkbox)] deep
    [INPUT (text)] deepm
    [INPUT (checkbox)] consto
    [INPUT (checkbox)] constn
    [INPUT (checkbox)] consta
    [INPUT (checkbox)] consts
    [INPUT (checkbox)] constb
    [INPUT (text)] limag
    [INPUT (checkbox)] starn
    [INPUT (text)] starnm
    [INPUT (checkbox)] starb
    [INPUT (text)] starbm
    [INPUT (checkbox)] flip
    [INPUT (text)] imgsize
    [INPUT (text)] fontscale
    [SELECT (select-one)] scheme
    [INPUT (checkbox)] edump
    [TEXTAREA (textarea)] elements

[download]

Fishing for tips. Thanks, Corion for your response and this considerable achievement:

$ wc -l $(locate Chrome.pm)
  5761 /home/hogan/Documents/repos/wmc/WWW-Mechanize-Chrome/lib/WWW/Me
+chanize/Chrome.pm
  5708 /usr/local/share/perl/5.26.1/WWW/Mechanize/Chrome.pm
 11469 total
$

[download]

Replies are listed 'Best First'.
Re^3: running an example script with WWW::Mechanize* module
by Corion (Patriarch) on Apr 19, 2020 at 06:28 UTC

    There is a misunderstanding of $mech->success - this method only reflects whether the last HTTP response from the server is considered an error or not. It does not reflect whether the last operation on $mech was successful or not. Error checking is usually done by die by WWW::Mechanize::Chrome.

    I haven't run your code, but the log output suggests that the form you're looking at has no name:

      say $mech->current_form->{name};    # ??
  # Use of uninitialized value in say at ./5.pluto.pl line 55.

    [download]

    The form is not great, because it really contains three fields with the same name date, so you will have to fetch the individual fields and explicitly set them:

    # largely untested
my @date_fields = $mech->selector('.//*[@name="date"]', node => $self-
+>current_form );
$mech->set_field( $date_fields[1] => $guess );

    [download]

    In the next version, I'll actually implement the arrayref form of ->set_fields() for values of index larger than one :) But that means breaking my (incompatible) API to restore the WWW::Mechanize API so I'll have to look carefully there.

     $mech->set_fields( $name => [ 'foo', 2 ] );

    [download]
[reply]
[d/l]
[select]
      The form is not great, because it really contains three fields with the same name date, so you will have to fetch the individual fields and explicitly set them:

      Thanks, Corion, I think we're almost there. I've got this pared down as far as I can to make an SSCCE. I can't get perl to think I have a valid selector:

      $ ./6.1.pluto.pl 
say (...) interpreted as function at ./6.1.pluto.pl line 47.
2020/04/20 20:10:39 Connected to ws://127.0.0.1:40757/devtools/browser
+/10f7d706-4dbd-4073-a438-916f6602fb4c
found the one and only form
Invalid rule, couldn't parse '//*[@name="date", node => $mech->current
+_form ]' at /usr/local/share/perl/5.26.1/HTML/Selector/XPath.pm line 
+283.
$

      [download]

      Source, with the critical line tried several different ways:

      #! /usr/bin/perl
use warnings;
use strict;
use WWW::Mechanize::Chrome;
use Log::Log4perl qw(:easy);
use Data::Dump;
use 5.016;

Log::Log4perl->easy_init($INFO);

my $site =
'https://www.fourmilab.ch/cgi-bin/Yoursky?z=1&lat=45.5183&ns=North&lon
+=122.676&ew=West';
my $mech = WWW::Mechanize::Chrome->new( headless => 1, );
$mech->get($site);

my $guess = 2458960;    #Earth day 2020 in julian days

$mech->form_number(1);
say "found the one and only form"
  if $mech->success();

# my best guess...aka...trial 1
# $mech->field( date => '2', jd => $guess );
# stderr: 3 elements found for input with name 'date' at ./6.1.pluto.p
+l line 21.

# your first guess...aka trial 2
# my @date_fields =$mech->selector( './/*[@name="date"]', node => $sel
+f->current_form );
# stderr: Global symbol "$self" requires explicit package name 

# 3rd guess
#my @date_fields =$mech->selector( './/*[@name="date"]', node => $mech
+->current_form );

# stderr: Invalid rule, couldn't parse '//*[@name="date"]' at /usr/loc
+al/share/perl/5.26.1/HTML/Selector/XPath.pm line 283.

# 4th guess
# my @date_fields =$mech->selector( './/*[@name="date", node => $self-
+>current_form ]');
# stderr: Invalid rule, couldn't parse '//*[@name="date", node => $sel
+f->current_form ]' 

# 5th guess
my @date_fields =$mech->selector( './/*[@name="date", node => $mech->c
+urrent_form ]');

# Invalid rule, couldn't parse '//*[@name="date", node => $mech->curre
+nt_form ]'

$mech->set_field( $date_fields[1] => $guess );

$mech->click_button( value => "Update" );    # this seems similar to W
+M
say "clickbutton succeeded"
  if $mech->success();
my $string = $mech->uri;
say ("We are at $string")
  if $mech->success();

      [download]

      That lays it out there as starkly as I can. VielenDank und Gruss aus Amiland.

[reply]
[d/l]
[select]
        Invalid rule, couldn't parse '//*[@name="date", node => $mech->current
+_form ]'

        [download]

        This looks as if you're mixing Perl code and an XPath expression in the same string.

         my @date_fields =$mech->selector( './/*[@name="date"]', node => $mech
+->current_form );
# stderr: Invalid rule, couldn't parse '//*[@name="date"]' at /usr/loc
+al/share/perl/5.26.1/HTML/Selector/XPath.pm line 283.

        [download]

        This is weird - I think this should still parse as an XPath expression. Does the following work and give the expected result?

         my @date_fields =$mech->xpath( './/*[@name="date"]', node => $mech->c
+urrent_form );

        [download]

        With version 0.48 of WWW::Mechanize::Chrome, ->set_fields now also supports selecing input elements by their name and index:

        $mech->set_fields(
    date => [3, $date_value],
);

        [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom