using POST in HTTP::Lite

deadpickle has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to send a POST to NCDC to get data from them. Not sure if this script is what I want. Does this script send anything to the URL or should i use a different module to do this?

#!/usr/bin/perl -w
use strict;
use HTTP::Lite;

my $req;
my $ncdc = new HTTP::Lite;
my $format = 'NEXRAD Level II';
my $dataset = '6500';
my $radar_site = 'KOAX';
my $year = '2001';
my $email = 'deadpickle@gmail.com';

my %vars = ("stations" => $radar_site,
            "station_lst" => "",
            "typeofdata" => "RADAR",
            "dtypelist" => "",
            "begdatestring" => "",
            "enddatestring" => "",
            "begyear" => $year,
            "begmonth" => '01',
            "begday" => '01',
            "beghour" => "",
            "begmin" => "",
            "endyear" => $year,
            "endmonth" => '12',
            "endday" => '31',
            "endhour" => "",
            "endmin" => "",
            "timeselectminlist" => "",
            "timeselectbuffer" => "",
            "gvartype" => "",
            "outmed" => "FTP",
            "outpath" => "",
            "pri" => "500",
            "altdsname" => "",
            "altstations1" => "",
            "altstations2" => "",
            "datasetname" => $dataset,
            "emailadd" => $email,
            "directsub" => "N",
            "outdest" => "FILE",
            "applname" => "",
            "subqueryby" => "STATION",
            "tmeth" => "Awaiting-Data-Transfer",
            );
            
    $ncdc->prepare_post(\%vars);
    $req = $ncdc->request("http://has.ncdc.noaa.gov/pls/plhas/HAS.File
+Select") or die "Unable to get document: $!";
    print "req: $req\n";
    print $ncdc->body();
[download]

When I run the script I get:

req: 404
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
The requested URL /pls/plhas/HAS.FileSelect/ was not found on this ser
+ver.<P>
<HR>
<ADDRESS>Oracle-Application-Server-10g/10.1.3.1.0 Oracle-HTTP-Server S
+erver at h
as.ncdc.noaa.gov Port 80</ADDRESS>
</BODY></HTML>
[download]

in return. As far as I know there is nowhere else to send the data but the URL given above.

Comment on using POST in HTTP::Lite Select or Download Code

Replies are listed 'Best First'.
Re: using POST in HTTP::Lite by jettero (Monsignor) on Jun 17, 2009 at 02:11 UTC
Maybe you're better off using something heavy like WWW::Mechanize. Then you probably don't have to figure out why you're getting a 404 (hint: your url is probably wrong). UPDATE: yeah, the comments below seem to indicate the 404 happens when the server is AWOL or overloaded or something. -Paul	[reply]
Re^2: using POST in HTTP::Lite by ikegami (Patriarch) on Jun 17, 2009 at 16:05 UTC
I almost made the snippy comment "Did you actually read the error message?" as soon as the OP was posted, but I tested it first. The url is correct. The page seems to send a 404 when it doesn't see the args it expects.	[reply]
Re: using POST in HTTP::Lite by Khen1950fx (Canon) on Jun 17, 2009 at 03:31 UTC
I think that the problem was that the HDSS system isn't currently available:-). This worked: #!/usr/bin/perl -w use strict; use HTTP::Lite; my $req; my $ncdc = new HTTP::Lite; my $format = 'NEXRAD Level II'; my $dataset = '6500'; my $radar_site = 'KOAX'; my $year = '2001'; my $email = 'deadpickle@gmail.com'; my %vars = ("stations" => $radar_site, "station_lst" => "", "typeofdata" => "RADAR", "dtypelist" => "", "begdatestring" => "", "enddatestring" => "", "begyear" => $year, "begmonth" => '01', "begday" => '01', "beghour" => "", "begmin" => "", "endyear" => $year, "endmonth" => '12', "endday" => '31', "endhour" => "", "endmin" => "", "timeselectminlist" => "", "timeselectbuffer" => "", "gvartype" => "", "outmed" => "FTP", "outpath" => "", "pri" => "500", "altdsname" => "", "altstations1" => "", "altstations2" => "", "datasetname" => $dataset, "emailadd" => $email, "directsub" => "N", "outdest" => "FILE", "applname" => "", "subqueryby" => "STATION", "tmeth" => "Awaiting-Data-Transfer", ); $ncdc->prepare_post(\%vars); $req = $ncdc->request("http://www.ngdc.noaa.gov/metadata/published +/NCDC/C00345.xml") or die "Unable to get document: $!"; print "req: $req\n"; print $ncdc->body(); [download]	[reply] [d/l]
Re^2: using POST in HTTP::Lite by deadpickle (Pilgrim) on Jun 17, 2009 at 16:01 UTC
HDSS is back up and running but I am still getting the error. How I see it is that request just requests a page, its not sending the post to anything. The reason I think this is because when I go to the page and pick data and click the submit button, the page moves on. But if request works the way I think it does then when you go to that url you get an error page. Is this wrong? UPDATE: Looking at WWW::Mechanize it seems like it may work well. I tried to install the module via CPAN shell but it is stuck on the cookie.t test. Any ideas how to get past it?	[reply]


No such thing as a small change
	PerlMonks