No this "endian" stuff always matters. Sorry to disappoint you. It is a "bummer" but that is the way that it is.

In order to write the Perl code, I would have to know the order of the byte stream in the file and then how that should look in memory.

Another way to do this if I know the input format, is to write a simple 'C' (or Perl) program for you that translates all these numbers into text as a pre-processing step. Then you just read these numbers in. For you, that avoids this "endian" stuff! The binary to text conversion has to know, but your code wouldn't. But sounds like your code would have to be able to handle 64 bit ints.