Id just like to point out that this is some scary code. Duplication like you have in that first section just screams out "refactor me!" You should convert as much of that into a subroutine, and then call it twice with the appropriate values. A perfect example of why you should do this are the following two lines

$s_ftp->binary() or die $s_ftp->message;
$d_ftp->binary() or die $s_ftp->message;
[download]

while you were cutting an pasting you forgot to change the second error condition from $s_ftp to $d_ftp. If I were writing it the code would probably look something like this:

use strict;
use warnings;
use Carp;

sub die_msg {
    my $ftp=shift;
    Carp::confess join "",@_,":\n",$ftp->message;
}

sub do_connect {
    my ($user,$pass,$host,$dir)=@_;

    my $ftp=Net::FTP->new($host)
        or die "Failed to connect to host '$host':$@";
    $ftp->login($user,$pass)
        or die_msg $ftp,"Failed login";
    $ftp->binary()
        or die_msg $ftp,"Failed to switch to binary mode";
    $ftp->cwd($dir)
        or die "Cant cwd '$dir'";

    return $ftp;
}


my $source_ftp=do_connect('user1','mypassword','some-server.com','/hom
+e/test/source');
my $dest_ftp  =do_connect('user2','mypassword','some-server.com','/hom
+e/test/dest');

my $source_file = "test.txt";
my $dest_file ||= $source_file;

$source_ftp->pasv_xfer($source_file, $dest_ftp, $dest_file)
  or warn "Failed xfer:\n" .
      $source_ftp->ok ? $dest_ftp->message : $source_ftp->message;
[download]

(Yes I know this doesnt address your real question. But this is also good advice. :-)