What I don't understand is WHY, since it's in a foreach loop first

The way I read your code, the (second) while loop is inside of the foreach loop. Snipping out the rest, the basic loop structure of your code looks like this:

while ( $offset < length( $pieces ) ) {
  # do some stuff
}
# do some more stuff
foreach my $p ( @pieces ) {
  # do some more stuff
  print "[dothis:] Piece $counted: $p \n";
  # do some more stuff
  while ($n = read (F, $data, $piece_length)  != 0) { 
     # yet more stuff
     print "[dothat] Hash2:" .  sha1_hex($data) . "\n\n";
     # yet more stuff
  }
}
[download]

So for each value of $p in the foreach loop, the inner while loop is going to run until it can't read any more data from F. On the first time through the foreach loop, this will read all the data from F that's available. On subsequent times through the foreach loop, all the data from F have already been read.