After much pondering, I think the problem is that you are initializing your bit vectors like this:

my $rin = 0;
[download]

but you should be doing this

my $rin = '';
[download]

When you say: $rin = 0; vec($rin, $fd, 1) = 1;, apparently the 0 is converted to the string "0" before the bit for $fd is set. This means that an extra bit will be set in each of your vectors.

Personally, I've never understood vec() well enough to want to use it. I always use IO::Select and can_read() rather than that. I highly recommend the IO::Socket and IO::Select methods because they are so easy to use and, I think, less error-prone.