How to get a better error message from LWP::UserAgent on missing SSL certificates?

moritz has asked for the wisdom of the Perl Monks concerning the following question:

I just spent several hours debugging some code using LWP::UserAgent. The code could be reduced to this:

use strict;
use warnings;
use LWP::UserAgent;

my $URL = 'https://myhost.local/';

my $ua = LWP::UserAgent->new;
$ua->ssl_opts( verify_hostname => 0 );
my $response = $ua->get($URL);

if ($response->is_success) {
    print $response->decoded_content;  # or whatever
}
else {
    die $response->status_line;
}
[download]

So pretty much a copy&paste from the LWP::UserAgent docs.

The error message I got was:

Can't connect to myhost.local:443\n\n
500 Can't connect to myhost.local:443 at foo.pl line 15.
[download]

After some debugging, strace finally showed me an ENOENT for an SSL CA file, which nudged me into the right direction.

So, my questions are:

Why does a missing CA file cause the request to fail, even though I've used ua->ssl_opts( verify_hostname => 0 );?
What do I have to do to get a better error message out of LWP::UserAgent? Is this a bug in LWP::UserAgent?

This is on a Debian Jessie box with perl 5.20.2, and LWP::UserAgent 6.06.

I'm writing a book on automating deployments and Continuous Delivery. Feedback very welcome!

Comment on How to get a better error message from LWP::UserAgent on missing SSL certificates? Select or Download Code

Replies are listed 'Best First'.
Re: How to get a better error message from LWP::UserAgent on missing SSL certificates? by hippo (Bishop) on Aug 31, 2016 at 10:52 UTC
Why does a missing CA file cause the request to fail, even though I've used `ua->ssl_opts( verify_hostname => 0 );`? Just because you aren't verifying the hostname doesn't mean it won't try to verify the certificate. The CA file is used to verify the certificate which may or may not match the hostname. What do I have to do to get a better error message out of LWP::UserAgent? Very difficult to get the precise problem as this is handled by the lower-level SSL modules. I'd recommend that you add `use IO::Socket::SSL 'debug4';` before you use LWP. That gives loads of output and should at least narrow down the problem to the SSL handshake if nothing more specific. I agree that this is very annoying and would be keen to see better diagnostics passed back up the tree to LWP and thus available to the users. That likely won't be easy.	[reply] [d/l] [select]
Re: How to get a better error message from LWP::UserAgent on missing SSL certificates? by Corion (Patriarch) on Aug 31, 2016 at 11:48 UTC
noxxi has written some tools for checking and debugging SSL connections. In his talk about debugging SSL, he especially recommends analyze-ssl.pl. For those who prefer spoken word over text, there also is a live talk by Steffen Ullrich on debugging SSL/TLS from the GPW 2015.	[reply]
Re: How to get a better error message from LWP::UserAgent on missing SSL certificates? by Krambambuli (Curate) on Aug 31, 2016 at 10:50 UTC
Just to make sure: was your https server working properly otherwise, could you connect with a browser against the same URL?	[reply]
Re^2: How to get a better error message from LWP::UserAgent on missing SSL certificates? by moritz (Cardinal) on Aug 31, 2016 at 11:35 UTC
I verified it with `curl -s $URL`, and it worked. Same with `wget --no-check-certificate $URL`. I'm writing a book on automating deployments and Continuous Delivery. Feedback very welcome!	[reply] [d/l] [select]
Re^3: How to get a better error message from LWP::UserAgent on missing SSL certificates? by Krambambuli (Curate) on Aug 31, 2016 at 13:03 UTC
I've tried to reproduce the problem, but it just works here. If I hide the CA certificate, I get an error with verify_hostname set, I don't if it is unset or set to 0. (Perl 5.22.2, LWP::UserAgent 6.15, LWP::Protocol::https 6.06 on Fedora 24.)	[reply]
Re^4: How to get a better error message from LWP::UserAgent on missing SSL certificates? by Krambambuli (Curate) on Aug 31, 2016 at 13:12 UTC
Re: How to get a better error message from LWP::UserAgent on missing SSL certificates? by Anonymous Monk on Aug 31, 2016 at 09:53 UTC
Dump the headers. LWP adds extra headers for more diagnostics, they start with Client	[reply]
Re^2: How to get a better error message from LWP::UserAgent on missing SSL certificates? by moritz (Cardinal) on Aug 31, 2016 at 10:37 UTC
I did: `Client-Date: Wed, 31 Aug 2016 08:58:47 GMT Client-Warning: Internal response` [download] Not very helpful. I'm writing a book on automating deployments and Continuous Delivery. Feedback very welcome!	[reply] [d/l]

Back to Seekers of Perl Wisdom