I don't think you are dealing with a Perl issue here, not directly. But just in case, add this at the start of your script to make your life easier and help us get to the root of the problem:

use strict;
use warnings;
use diagnostics;
[download]

You are running this on a Raspberry Pi, right? In my experience, this hardware had a lot of "unexpected features", like messing up SPI timing when a new ethernet packet arrives.

In your case, you seem to transfer more than 16 bytes at one time, but i rather suspect the serial hardware has a 16 byte limited buffer, and that is most likely where your timing gets messed up and you drop some bytes. This happens when the system is busy with other stuff (like dealing with ethernet or other interrupts) and can't get to working on the serial buffer in time.

You have quite a few alternatives here. You could use another hardware or try one of those serial-to-ethernet boxes. You could also try a different Serial-to-USB adapter.