Random_Walk has asked for the wisdom of the Perl Monks concerning the following question:
Most esteemed monks,
I have a script pulling back CMDB info from service now using REST API. works a treat on my windows laptop where developed. Move it to Linux and I get a 400 back. I have cut things all the way back and found it is not REST issue, but even a basic LWP get is failing. Here is my test script that work on win, fails on lin.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->proxy('https', $ENV{HTTPS_PROXY});
my $response = $ua->get('https://google.com/');
if ($response->is_success) {
print $response->decoded_content; # or whatever
}
else {
die $response->status_line;
}
Here is the curl test
myaccount@linuxhost:~> curl https://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset
+=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.se/?gfe_rd=cr&ei=XwaLWMjfB6Sr8weh06_QD
+Q">here</A>.
</BODY></HTML>
My HTTPS_PROXY is set, and I can pull back https://google.com with curl. What magic do I miss?
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by Corion (Patriarch) on Jan 27, 2017 at 09:10 UTC
|
Consider looking at the actual data that gets exchanged via LWP::ConsoleLogger maybe.
I assume that your proxy wants a username+password, which you don't give it. Maybe curl is better or uses some different credentials there.
| [reply] |
|
me@linux:/home/rpollard> export HTTPS_PROXY='http://user:pass@proxy:80
+80'
me@linux:/home/rpollard> echo $HTTPS_PROXY
http://user:pass@proxy:8080
me@linux:/home/rpollard> curl https://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset
+=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.se/?gfe_rd=cr&ei=6T-MWJGBLsiq8weZ_YCIA
+Q">here</A>.
</BODY></HTML>
me@linux:/home/rpollard> perl GetTest.pl
400 Bad Request at GetTest.pl line 16.
me@linux:/home/rpollard> unset HTTPS_PROXY
me@linux:/home/rpollard> echo $HTTPS_PROXY
me@linux:/home/rpollard> curl https://www.google.com
curl: (6) Could not resolve host: www.google.com
me@linux:/home/rpollard> perl GetTest.pl
500 Can't connect to google.com:443 (Bad hostname) at GetTest.pl line
+16.
me@linux:/home/rpollard> perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-li
+nux-thread-multi
Copyright 1987-2013, Larry Wall
Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.
Complete documentation for Perl, including FAQ lists, should be found
+on
this system using "man perl" or "perldoc perl". If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.
That is using the same test script as in the root node. I think this shows both curl and perl are using the same HTTPS_PROXY here. Now I have a more modern Perl, I will install LWP::ConsoleLogger and see what that tells me.
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] [d/l] |
|
Server is all locked down, no gcc/cc etc and seos won't even let me install gcc, so I can't cpan LWP::ConsoleLogger. Now I can either go through the full change cycle to get a test server, or use curl in my scripts to pull back the data form the REST API I am after, then parse it from there. Neither choice appeals
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] [d/l] |
|
cpan LWP::Protocol::https
This http://blogs.perl.org/users/brian_d_foy/2011/07/now-you-need-lwpprotocolhttps.html brought me enlightenment.
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] [d/l] |
|
| [reply] |
|
Hi Corion,
I have HTTP_PROXY and HTTPS_PROXY set with http://user:pass@proxyhost and these are used by curl so must be correct.
I tried to install lwp::ConsoleLogger with CPAN, and it failed as my Perl is too old (5.10). I suspect this may be the cause, so I am having a quick look around for a server in the same data center with a newer Perl to test if that is the critical difference. The Perl on my lappy that works is 5.22.1.
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] |
|
# curl https://google.com/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset
+=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.se/?gfe_rd=cr&ei=IvWhWPTvI4b67gSTn7ygA
+g">here</A>.
</BODY></HTML>
# perl test.pl
GET https://google.com/
.-----------------+--------------------.
| Request Header | Value |
+-----------------+--------------------+
| Accept-Encoding | gzip |
| User-Agent | WWW-Mechanize/1.83 |
'-----------------+--------------------'
==> 500 Can't connect to google.com:443
.-----------------+-------------------------------.
| Response Header | Value |
+-----------------+-------------------------------+
| Client-Date | Mon, 13 Feb 2017 18:04:23 GMT |
| Client-Warning | Internal response |
| Content-Type | text/plain |
'-----------------+-------------------------------'
.---------------------------------------------------------------------
+---------------------------------------------------------------------
+---.
| Content
+
+ |
+---------------------------------------------------------------------
+---------------------------------------------------------------------
+---+
| Can't connect to google.com:443
+
+ |
|
+
+ |
| Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro
+tocol/http.pm line 47, <DATA> line 1.
+ |
'---------------------------------------------------------------------
+---------------------------------------------------------------------
+---'
.---------------------------------------------------------------------
+---------------------------------------------------------------------
+---.
| Text
+
+ |
+---------------------------------------------------------------------
+---------------------------------------------------------------------
+---+
| Can't connect to google.com:443
+
+ |
|
+
+ |
| Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro
+tocol/http.pm line 47, <DATA> line 1.
+ |
'---------------------------------------------------------------------
+---------------------------------------------------------------------
+---'
Error GETing https://google.com/: Can't connect to google.com:443 at t
+est.pl line 22.
Thanks for taking the time. I have at least learned a bit more about the web tools available in Perl :)
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] [d/l] |
|
Unfortunately, LWP::ConsoleLogger doesn't seem to be talkative about when it is trying to connect to a proxy.
You can try to find which proxy LWP will try to use:
#!perl -w
use strict;
use LWP::ConsoleLogger::Easy 'debug_ua';
use Data::Dumper;
use LWP::UserAgent;
my $ua=LWP::UserAgent->new();
warn Dumper $ua->proxy('https');
my $logger=debug_ua($ua,5);
print Dumper \%ENV;
$ua->env_proxy;
warn Dumper $ua->proxy('https');
$ua->get('https://google.com')
This should show you that your proxy URL is in %ENV and also picked up from there in the $ua.
Maybe your proxy does not let Perl through? Have you tried masquerading your Perl script as curl (because that seems to work?)?:
$ua->default_headers->header('User-Agent' => 'curl/7.10.x (i386-redhat
+-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4');
| [reply] [d/l] [select] |
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by shadowsong (Pilgrim) on Jan 27, 2017 at 15:09 UTC
|
Hi R
As Anonymous said above… what do you get when you try: lwp-request -m get -USsed http://ppm.activestate.com/CPAN/authors http://cpan.perl.org/authors?
When I do it - I get:
C:\Users\shadowsong>lwp-request -m get -USsed http://ppm.activestate.
+com/CPAN/authors http://cpan.perl.org/authors
GET http://ppm.activestate.com/CPAN/authors
User-Agent: lwp-request/6.15 libwww-perl/6.15
407 authenticationrequired
Cache-Control: no-cache
Date: Fri, 27 Jan 2017 15:05:31 GMT
Via: 1.1 10.156.69.200 (McAfee Web Gateway 7.6.2.21509)
Proxy-Authenticate: NTLM
Content-Length: 3895
Content-Type: text/html
Client-Date: Fri, 27 Jan 2017 15:05:31 GMT
Client-Peer: 10.156.69.200:3128
Client-Response-Num: 1
Client-Warning: Unsupported authentication scheme 'ntlm'
Link: </mwg-internal/de5fs23hu73ds/files/default/stylesheet.css>; rel=
+"stylesheet"
Proxy-Connection: Keep-Alive
Title: McAfee Web Gateway - Notification
X-UA-Compatible: IE=7
GET http://cpan.perl.org/authors
User-Agent: lwp-request/6.15 libwww-perl/6.15
407 authenticationrequired
Cache-Control: no-cache
Date: Fri, 27 Jan 2017 15:05:31 GMT
Via: 1.1 10.156.69.200 (McAfee Web Gateway 7.6.2.21509)
Proxy-Authenticate: NTLM
Content-Length: 3873
Content-Type: text/html
Client-Date: Fri, 27 Jan 2017 15:05:31 GMT
Client-Peer: 10.156.69.200:3128
Client-Response-Num: 1
Client-Warning: Unsupported authentication scheme 'ntlm'
Link: </mwg-internal/de5fs23hu73ds/files/default/stylesheet.css>; rel=
+"stylesheet"
Proxy-Connection: Keep-Alive
Title: McAfee Web Gateway - Notification
X-UA-Compatible: IE=7
Shadowsong | [reply] [d/l] [select] |
|
me@linuxbox:~> lwp-request -m get -USsed http://ppm.activestate.com/C
+PAN/authors http://cpan.perl.org/authors?
GET http://ppm.activestate.com/CPAN/authors/
User-Agent: lwp-request/5.810
GET http://ppm.activestate.com/CPAN/authors --> 301 Moved Permanently
GET http://ppm.activestate.com/CPAN/authors/ --> 200 OK
Cache-Control: proxy-revalidate
Connection: close
Date: Fri, 27 Jan 2017 16:11:05 GMT
Age: 0
Server: Apache
Content-Length: 4728
Content-Type: text/html;charset=UTF-8
Client-Date: Fri, 27 Jan 2017 16:11:53 GMT
Client-Peer: 10.244.0.50:83
Client-Response-Num: 1
Title: Index of /CPAN/authors
GET http://www.cpan.org/authors/?
User-Agent: lwp-request/5.810
GET http://cpan.perl.org/authors? --> 301 Moved Permanently
GET http://www.cpan.org/authors? --> 301 Moved Permanently
GET http://www.cpan.org/authors/? --> 200 OK
Cache-Control: proxy-revalidate
Connection: close
Date: Fri, 27 Jan 2017 16:11:54 GMT
Via: 1.1 varnish-v4
Via: 1.1 varnish
Via: 1.1 varnish
Accept-Ranges: bytes
Age: 0
Server: Apache/2.2.15 (CentOS)
Vary: Accept-Encoding
Content-Length: 4722
Content-Type: text/html;charset=UTF-8
Client-Date: Fri, 27 Jan 2017 16:11:54 GMT
Client-Peer: 10.244.0.50:83
Client-Response-Num: 1
Fastly-Debug-Digest: 492dd663e1697be6e60aeecd19b5bf028765c1b3d66167d36
+df53fb8eefe26c0
Title: Index of /authors
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Served-By: cache-lax8636-LAX, cache-bma7023-BMA
X-Timer: S1485533514.441262,VS0,VE504
Now if I go for https://google.com
me@linuxbox:~> lwp-request -m get -USsed https://google.com
GET https://google.com
User-Agent: lwp-request/5.810
GET https://google.com --> 400 Bad Request
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Content-Length: 677
Content-Type: text/html; charset=utf-8
Client-Date: Fri, 27 Jan 2017 16:16:23 GMT
Client-Peer: 10.244.0.50:83
Client-Response-Num: 1
Proxy-Connection: close
Title: Request Error
At this site I have the same proxt settings for http and https, which strikes me as odd, but works fine on my windows lappy.
Cheers, R.
Pereant, qui ante nos nostra dixerunt!
| [reply] [d/l] [select] |
|
HTTP_PROXY_USER=your_proxy_username
HTTP_PROXY_PASS=your_proxy_password
And update HTTP_PROXY to:
HTTP_PROXY=http://proxy.my.net:8080
Best of luck! Shadowsong
| [reply] [d/l] [select] |
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by Anonymous Monk on Jan 27, 2017 at 09:00 UTC
|
| [reply] |
Re: HTTPS returning 400 from Linux but 200 OK from Windows
by madtoperl (Hermit) on Jan 27, 2017 at 14:32 UTC
|
$ua->credentials( $linktologin, $pause, $username, $password );
Eg. $ua->credentials( 'https://google.com/:443', 'PAUSE', ‘$username
+’, ‘$password’);
You can refer below link for more details, https://perlmaven.com/lwp-useragent-and-basic-authentication
| [reply] [d/l] |
|
| [reply] [d/l] |
|
|