Help with LWP & SSL 500 Internal response error

dcsastry has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I have a vexing problem with Perl LWP and accessing an internal company HTTPS site.

I was having lots of problems getting an SSL handshake to work, and I finally got around that.

Then I was having problems with credentials not working, and I finally got around that.

Now I am having some kind of internal issue with HTTP. It seems like the SSL handshake is done, but I cannot pull data from the website after that. I just get this 500 internal error on the client side.

I am not sure what I can do to debug this issue - any ideas for things I need to look at? Thanks!

Here is the code I used (edited for posting)

use LWP::UserAgent;
use LWP::Protocol::https;
use HTTP::Request;
use HTTP::Cookies;

$IO::Socket::SSL::DEBUG=3;

our $ua = LWP::UserAgent->new(keep_alive=>1);
$ua->agent("Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11
+.0) like Gecko");
$ua->ssl_opts(SSL_ca_file => 'C:\OpenSSL-Win32\cert\companyx Root CA B
+ase64.cer');
$ua->ssl_opts(verify_hostname=>1);
$ua->cookie_jar(HTTP::Cookies->new(file => "$ENV{HOME}/.cookies.txt"))
+;
$ENV{http_proxy} = '';
our $host = "cdis.companyx.com:9283";
our $url1 = "https://$host/cdis/default.asp";
$ua->credentials("$host",'', "AMR\\username", 'password');
$ua->timeout(10);
my $response1 = $ua->get($url1);
if ($response1->is_success) {
    print $response1->content;
} else {
    print "Error: " . $response1->status_line . "\n";
    print $response1->headers()->as_string(), "\n";
    print $response1->content;
}
exit(0);
[download]

Here is the output

DEBUG: .../IO/Socket/SSL.pm:2633: new ctx 62187576
DEBUG: .../IO/Socket/SSL.pm:549: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:551: socket connected
DEBUG: .../IO/Socket/SSL.pm:573: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:606: using SNI with hostname cdis.companyx
+.com
DEBUG: .../IO/Socket/SSL.pm:641: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:660: set socket to non-blocking to enforce
+ timeout=10
DEBUG: .../IO/Socket/SSL.pm:676: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:686: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:696: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:716: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2536: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2489: ok=1 cert=62234208
DEBUG: .../IO/Socket/SSL.pm:2489: ok=1 cert=62234544
DEBUG: .../IO/Socket/SSL.pm:2489: ok=1 cert=62234432
DEBUG: .../IO/Socket/SSL.pm:2489: ok=1 cert=62234320
DEBUG: .../IO/Socket/SSL.pm:1581: scheme=www cert=62234320
DEBUG: .../IO/Socket/SSL.pm:1591: identity=cdis.companyx.com cn=cdis.c
+ompanyx.com alt=2 fm1seagwebpd01.amr.corp.companyx.com 2 fm1seagwebpd
+02.amr.corp.companyx.com 2 fm1seagwebpd03.amr.corp.companyx.com 2 cdi
+sprod-node1.companyx.com 2 cdisprod-node2.companyx.com 2 cdisprod-nod
+e3.companyx.com 2 cdis.companyx.com
DEBUG: .../IO/Socket/SSL.pm:676: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:686: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:696: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:716: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:676: Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:731: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2666: free ctx 62187576 open=62187576
DEBUG: .../IO/Socket/SSL.pm:2671: free ctx 62187576 callback
DEBUG: .../IO/Socket/SSL.pm:2678: OK free ctx 62187576
Error: 500 Status read failed: Unknown error
Content-Type: text/plain
Client-Date: Tue, 12 May 2015 15:19:46 GMT
Client-Warning: Internal response

Status read failed: Unknown error at C:/Perl/site/lib/Net/HTTP/Methods
+.pm line 276.
[download]

Comment on Help with LWP & SSL 500 Internal response error Select or Download Code

Replies are listed 'Best First'.
Re: Help with LWP & SSL 500 Internal response error by boftx (Deacon) on May 12, 2015 at 15:58 UTC
Have you checked the server log to see what the internal error is? You must always remember that the primary goal is to *drain the swamp* even when you are hip-deep in alligators.	[reply]
Re^2: Help with LWP & SSL 500 Internal response error by dcsastry (Initiate) on May 12, 2015 at 17:03 UTC
Unfortunately I don't have access to the server log. This is a internal company site that I do not manage - I am just trying to access specific pages of the site, but it requires an HTTPS connection.	[reply]
Re^3: Help with LWP & SSL 500 Internal response error by afoken (Chancellor) on May 12, 2015 at 17:22 UTC
`Error: 500 Status read failed: Unknown error Content-Type: text/plain Client-Date: Tue, 12 May 2015 15:19:46 GMT Client-Warning: Internal response` [download] This error was generated on the client side by LWP::UserAgent (see last line), not by the server. It seems the communication with the HTTPS server failed. Are you sure your LWP installation supports HTTPS? Try connecting to https://www.google.com/ or similar. Maybe the HTTPS server uses a self-signed certificate? (But that should give a different error message ...) Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l]
A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Help with LWP & SSL 500 Internal response error by noxxi (Pilgrim) on May 12, 2015 at 20:15 UTC
According to the log the SSL handshake itself succeeded: DEBUG: .../IO/Socket/SSL.pm:676: Net::SSLeay::connect -> 1 DEBUG: .../IO/Socket/SSL.pm:731: ssl handshake done Which means the error must be at the HTTP layer. Check if the same URL is accessible with a browser, curl, wget ... - it can be that the server restricts access to specific user agents.	[reply]
Re: Help with LWP & SSL 500 Internal response error by Anonymous Monk on May 12, 2015 at 23:10 UTC
You should upgrade all the modules (LWP / IO::Socket::SSL / Net...) and try again :)	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks