I'm writing a podcatcher and I ran into a problem where one particular podcast (whose RSS feed downloads fine in Firefox or wget) was failing to download when I used the default HTTP headers I used with all the other podcasts I've tested with. It would give a "406 Not Aceptable" error, which research suggested was caused by bad Accept headers. I ran wget -d to see what headers it was using and copied them into my script (or rather the config file for my script), and got the same error with this podcast. Below, find a simplified version of the code that reproduces the problem.

#! /usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0');
$ua->show_progress( 1 );

my %headers = ( 'Accept' => '*/*',
        'Accept-Encoding' => 'identity',
        'Connection' => 'Keep-Alive',
#        'Host' => 'feed.podbean.com' );
        'Host' => 'makeoursmarvel.com' );

#my $url = 'https://feed.podbean.com/betheserpent/feed.xml';
my $url = 'http://makeoursmarvel.com/feed/podcast/';


my $response = $ua->get( $url, %headers );

if ( $response->is_success ) {
    print "Okay!\n";
} else {
    my $status = $response->status_line;
    print "failed to download $url: $status\n";
}
[download]

If I use the commented out lines instead for the Host and $url value (checking a different podcast's RSS feed), everything works fine. I also tried using the default Firefox Accept: header, based on https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values, and got the same 406 error. The same podcast also give me a 406 error when I try to get an individual mp3 file.

Any ideas how to narrow the problem down further, if not fix it?

I'm using Perl v5.26.1 and LWP::UserAgent version 6.31.