LWP::UserAgent post, file upload corruption.

FeistyLemur has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, so I have a bit of code that was previously working fine and for no reason I can find an explanation to has stopped working. Basically I am doing a firmware file upload to a device via http. I've used this code many times in the past, even on the same test device and now it fails with a corrupt file error message from the device. Below is the relevant snip for the file upload. It's a much larger script with other parts for configuration, so I've tried to just carve out the upload here for an example.

use LWP::UserAgent();
use HTTP::CookieJar::LWP();

## Create browser object.
my $jar = HTTP::CookieJar::LWP->new;
my $ua  = LWP::UserAgent->new(
    keep_alive=>200,
    cookie_jar => $jar,
    protocols_allowed => ['http', 'https'],
    timeout => 60,
);

my $client='xxx.xxx.xxx.xxx';
my $username='username';
my $password'password';
my $fw_file='/home/location/file.bin';
my $xsrf=auth($client, $username, $password); ##xsrf session token is 
+provided by separate authentication sub.
my $results=upgrade($client, $fw_file);

sub upgrade {
    ## Upload file.
    my ($client,$fw_file)=@_;    
    my $res=$ua->post("http://$client/upload.cgi?xsrf=$xsrf", [
        'upfile' => ["$fw_file"],
        'Submit_button' => 'Upload+Software+Image',
        ],
        Content_Type => 'form-data',
    );
    ##Continues on to other steps from here.
    ##This post fails where it worked previously. 
}
[download]

All the documentation and examples I've found have suggested posting file uploads to http in this way and it worked for ages, I would have to assume system package updates changed the way things are working, as the device being uploaded to has not changed in any way. Any thoughts about what might be handled better in this method?

Comment on LWP::UserAgent post, file upload corruption. Download Code

Replies are listed 'Best First'.
Re: LWP::UserAgent post, file upload corruption. by bliako (Monsignor) on Feb 04, 2020 at 12:48 UTC
Documentation for HTTP::Request::Common states: If a $file is provided by no Content-Type header, then Content-Type and Content-Encoding will be filled in automatically with the values returned by LWP::MediaTypes::guess_media_type() Maybe a module upgrade has changed the behaviour of `LWP::MediaTypes::guess_media_type()` to guess differently for `.bin` files. Try POSTing with the encoding set explicitly, like `'Content-Type' => 'application/octet-stream'`. To investigate further, use LWP in debug mode as follows and observe the headers: `use LWP::UserAgent(); use LWP::ConsoleLogger::Easy qw( debug_ua ); use HTTP::CookieJar::LWP(); ## Create browser object. my $jar = HTTP::CookieJar::LWP->new; my $ua = LWP::UserAgent->new( keep_alive=>200, cookie_jar => $jar, protocols_allowed => ['http', 'https'], timeout => 60, ); debug_ua($ua); # etc.` [download] bw, bliako	[reply] [d/l] [select]
Re^2: LWP::UserAgent post, file upload corruption. by FeistyLemur (Acolyte) on Feb 04, 2020 at 15:38 UTC
I did try setting the content type to application/octet-stream manually, I also tried using HTTP::Request::Common, and the DYNAMIC_FILE_UPLOAD option as per another guide I found. Neither helped. I will try using the debug mode next and see what that yields. .	[reply]
Re: LWP::UserAgent post, file upload corruption. by jcb (Parson) on Feb 04, 2020 at 00:31 UTC
Are you sure that `'/home/location/fie.bin'` has not been somehow corrupted? Do you have a way to verify that file other than uploading it to the test device?	[reply] [d/l]
Re^2: LWP::UserAgent post, file upload corruption. by FeistyLemur (Acolyte) on Feb 04, 2020 at 15:47 UTC
The file is good, I use this updater on several machines as well as my own which all have the same issue, I am able to upload and even upgrade to this exact file using the web interface of the device in a browser without error. The device checksums the file before going ahead with a firmware flash, so it's definitely not corrupt on the disk. Using wireshark the post never even appears (it does when using the web ui, and did using this script before), all I get is a 200 ok after attempting the post, with the error message that states "connection was interrupted, or the file is corrupt. As per the other reply I will try the LWP debug and see how that goes for finding more info."	[reply]


Perl: the Markov chain saw
	PerlMonks