Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

500 error with LWP:UserAgent

by RenardBleu (Sexton)
on Nov 04, 2014 at 09:56 UTC ( #1106004=perlquestion: print w/replies, xml ) Need Help??

RenardBleu has asked for the wisdom of the Perl Monks concerning the following question:

Solved !
thanks for every help i had :)

Found and solved my issue with
use Net::SSL (); # From Crypt-SSLeay BEGIN {<br /> $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL $ENV{HTTPS_PROXY} = ''; }
------------------ Hi
I tried a thousand different ways to connect to the URL and i seem unable to get something else than a 500 error.

Latest code i tried was
require LWP::UserAgent; my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0}); $ua->agent("Mozilla/8.0"); $ua->timeout(10); my $req = HTTP::Request->new( GET => " +s/entradaXMLEntidad/"); $req->header( 'Accept' => 'text/html' ); # send request my $res = $ua->request($req); # check the outcome if ( $res->is_success ) { print $res->decoded_content; } else { print "Error: " . $res->status_line . "\n"; }

I tried with and withcout $ua->proxy('https', '');
I tried POST and GET (since in the end, ill have to access that URL with POST)

of course I made sure I was able to get this page from a regular browser. I made sure my code always was able to connect to other pages with the form of https://some-domain:some-port/path/to/another/page

If anyone can help me to find what i do wrong, I d be thankfull

Replies are listed 'Best First'.
Re: 500 error with LWP:UserAgent
by Loops (Curate) on Nov 04, 2014 at 10:16 UTC

    Hi there!

    Would like to help, but this is not a Perl question. As you said, changing the URL to another site, it-just-works. The problem is knowing what that payment system company expects; unfortunately their documentation seems to be private. You'll need a peek at the documentation or some help from them directly.

    Update: Okay, have to apologize I guess, since I decided to try your code here anyway. It turns out with v5.18.4, i get your 500 error, and with v5.20.0 i get web content back saying that an error occurred (in Spanish). With the exact same copy of your code. Odd.

    Update2: Wireshark revealed it was an SSL error causing the problem. Something must have changed in the Perl SSL module between v18 and v20. Anyway, if you change the ssl_opt to that given below, the code works in either version:

    my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0, SSL_v +ersion => 'TLSv1' });

    You may want to look at the other SSL options as listed in the IO::Socket::SSL docs.

      Hi there,

      I tried your modification and it didnt change my case.

      I use
      perl v5.14.2
      LWP::UserAgent v6.06
      HTTP::Request v6.00
      IO::Socket::SSL 1.955

      use Net::SSLeay; $Net::SSLeay::trace = 2;

      as suggested on stackoverflow : produced the output
      DEBUG: .../IO/Socket/ socket not yet connected DEBUG: .../IO/Socket/ socket connected DEBUG: .../IO/Socket/ ssl handshake not started DEBUG: .../IO/Socket/ using SNI with hostname sis-t.redsys. +es DEBUG: .../IO/Socket/ set socket to non-blocking to enforce + timeout=10 DEBUG: .../IO/Socket/ ssl handshake in progress DEBUG: .../IO/Socket/ waiting for fd to become ready: SSL w +ants a read first DEBUG: .../IO/Socket/ handshake failed because socket did n +ot became ready Error: 500 Can't connect to

        Well it's a puzzle for sure. I installed Perl and those modules with the same version you have, and it just worked with the updated code. The old code gave the same Net::SSLeay debug output as yours. There is some underlying difference between our setups obviously.

        There's a chance that changing the SSL version to this would help:

        SSL_version => 'TLSv1:!TLSv11:!TLSv12:!SSLv2:!SSLv3'

        The last two components are supposedly redundant. But in any case, that long version string wasn't needed here (with any of 3 machines).

        Update: Just as a debugging step, try adding random garbage to the string you pass into SSL_version. You should expect the following error, otherwise something is up in Denmark:

        Error: 500 invalid SSL_version specified
Re: 500 error with LWP:UserAgent
by Discipulus (Abbot) on Nov 04, 2014 at 10:23 UTC
    yes, as stated seems not a problem with your code. Browsing the link you'll have:
    La forma en que intenta acceder al sistema de pago no es correcta. Este mensaje lo puede estar viendo por dos motivos: 1.- Nos envía el formulario con los datos por método GET. 2.- Está usted intentando navegar por esta página; esta pagina no está + accesible para entrar directamente desde el navegador de internet. Para acceder correctamente al TPV Virtual debe enviar a esta URL un fo +rmulario HTML mediante POST, con los parámetros descritos en la docum +entación que se obtiene a través del módulo de administración. Por favor, tenga en cuenta esto para poder realizar pagos por internet + con nuestro sistema.
    And using my, still not perfect, i got: -u +ad/ -v 4 -h ==================================================================== Tue Nov 4 11: +17:21 2014 -------------------------------------------------------------------- Response code: 404 Response message: Not Found Response server: UNDEF Response declared length: 219 Response title: 404 Not Found -------------------------------------------------------------------- main page content (1): 219.0000 bytes in 61.573974 seconds @ 3.5567 b +ytes/s) -------------------------------------------------------------------- detail of loaded pages (url): -------------------------------------------------------------------- -------------------------------------------------------------------- no included content found. no external content found. no broken links found. -------------------------------------------------------------------- downloaded 219.0000 bytes (219 bytes) in 61.573974 seconds (3.5567 byt +es/s) ==================================================================== headers received: -------------------------------------------------------------------- Connection: close Date: Tue, 04 Nov 2014 10:17:20 GMT Content-Length: 219 Content-Type: text/html; charset=iso-8859-1 Client-Date: Tue, 04 Nov 2014 10:17:21 GMT Client-Peer: Client-Response-Num: 1 Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Netwo +rk/OU=Terms of use at (c)10/CN=VeriSign Class 3 International Server CA - G3 Client-SSL-Cert-Subject: /C=ES/ST=Madrid/L=Madrid/O=REDSYS SERVICIOS D +E PROCESAMIENTO, S.L./OU=REDSYS SERVICIOS DE PROCE SAMIENTO, S.L./ Client-SSL-Cipher: AES128-SHA Client-SSL-Warning: Peer certificate not verified Set-Cookie: TS8e1dfd=a50bf1628fedf28fafe3b2ec1d8aca3ea874d6dd7c147bd15 +458a7b0; Path=/ Title: 404 Not Found ====================================================================

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
Re: 500 error with LWP:UserAgent
by Khen1950fx (Canon) on Nov 05, 2014 at 14:14 UTC
    Stick with POST. GET just gives you an "incorrect" warning; however, using POST will give you an exception. Here's the code that I ran:
    #!/usr/bin/perl -l use strict; use warnings; use Encode; use Encode::Locale; require LWP::UserAgent; require HTTP::Request; my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 } ); $ua->agent("Mozilla/8.0"); $ua->timeout(10); $ua->protocols_allowed( ['https'] ); my $req = HTTP::Request->new( 'POST', '' ); $req->header( 'Accept' => 'text/html' ); my $res = $ua->request($req); if ( $res->is_success ) { print $res->decoded_content; } else { print "Error: " . $res->status_line . "\n"; }
    And the result:
    <!-- RSisException --><html> <head> <META http-equiv="Content-Type" content="text/html; charset=ISO-8859-1 +"> <meta content="-1" http-equiv="Expires"> <meta content="Monday, 01-Jan-90 00:00:00 GMT" http-equiv="Expires"> <meta content="no-cache" http-equiv="Pragma"> <meta content="no-cache" http-equiv="Cache-Control"> <script type="text/javascript" src="/dtagent56_n3_5964.js" data-dtconf +ig="rid=RID_1948368250|rpid=272727326|tp=500,50,0|">< +/script><link type="text/css" href="/sis/estilos/unica/9999.css" rel= +"StyleSheet"> <link type="text/css" rel="StyleSheet" href="/sis/estilos/.css"> <script src="/sis/javascript/xsl/RSisException.js" type="text/javascri +pt"></script><script src="/sis/javascript/utilSis.js" type="text/java +script"></script> </head> <body onload="load()"> <div id="divCuerpo" style="overflow:auto;height:90%;"> <table class="contenedor"> <tr> <td height="52" colspan="2" align="right" class="tituloSuperior"><img +height="52" onerror="this.src='/sis/graficos/logotipos/comunes/redsys +.gif';" src="/sis/graficos/logotipos/comunes/-1.gif" alt="Terminal de + pagos virtuales"></td> </tr> <tr> <td colspan="2" class="separador">&nbsp;</td> </tr> <tr> <td colspan="2" class="resultado"><font class="denegacion"> <!--:--></font></td> </tr> <tr> <td colspan="2" class="separador">&nbsp;</td> </tr> <tr> <td colspan="2" class="botones"> <form action="" method="get"> <div style="position:relative;visibility:hidden" id="divImg"> <input type="button" class="botonAccion" onmouseover="hov(this,'botonA +ccion btnhov')" onmouseout="hov(this,'botonAccion')" value="Imprimir" + onClick="javascript:imprimir();" alt="Imprimir"><input type="button" + class="botonAccion" onmouseover="hov(this,'botonAccion btnhov')" onm +ouseout="hov(this,'botonAccion')" value="Cerrar" onClick="window.clos +e();" alt="Cerrar"> </div> <div style="position:relative;visibility:visible" id="divImgNoScript"> <NOSCRIPT value="Imprimir"> &nbsp; <input type="submit" class +="buttonAccion" value="Cerrar" ALT="Pulse el bot&oacute;n para cerrar +"> </NOSCRIPT> </div> </form> </td> </tr> </table> </div> <div id="divPie" style="overflow:visible; height:55px;"> <table cellpadding="0" cellspacing="0" border="0" width="100%"> <tr style="background-color:#8e9092;"> <td height="52" style="width:100%">&nbsp;</td> </tr> </table> </div> </body> </html>

    Make sure that you have these modules installed:

Re: 500 error with LWP:UserAgent
by noxxi (Pilgrim) on Nov 05, 2014 at 18:53 UTC

    > ...

    This server has some serious issues. If you use the old default cipher set of IO::Socket::SSL, that is "ALL:!LOW", with TLS1.0 or higher the connection simply hangs. This kind of problem is often seen in connection with older F5 load balancers in front of the server, which simply drop ClientHello packets greater than 255 bytes. Since version 1.962 (about a year ago) IO::Socket::SSL will use a smaller (and more secure) cipher set to work around such problems. But I guess you are using an older version of IO::Socket::SSL.

    Crypt::SSLeay instead uses the default cipher set of OpenSSL, which makes the packet in this case just 248 bytes and thus small enough. That's why it works with Crypt::SSLeay. But please note, that Crypt::SSLeay does not make any verification of the hostname against the servers certificate and is thus open to man-in-the-middle attacks.

    With current versions of IO::Socket::SSL the problem should be fixed too. But, since you are using a proxy you should use the latest versions of LWP::UserAgent and LWP::Protocol::https too, because proper proxy support with IO::Socket::SSL backend was only added to version 6.06 (it might be in 6.04 already if you use Debian or a spinoff like Ubuntu).

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1106004]
Approved by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2022-06-26 16:30 GMT
Find Nodes?
    Voting Booth?
    My most frequent journeys are powered by:

    Results (86 votes). Check out past polls.