http://qs321.pair.com?node_id=622081

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am working on a script to read and interpret a binary file, which consists of a variable number of variable length strings.
There are count values embedded in the file for the number of strings, and the length of each string, e.g (artificial example):
NumberOfStrings (1 byte) Str1Length (1 byte) Str1Characters (Str1Length bytes) Str2Length (1 byte) Str2Characters (Str2Length bytes) ... StrNLength (1 byte) StrNCharacters (StrNLength bytes)
The script itself contains something like this:
# Create an example byte-sequence $count = 3; $str1 = "First string"; $str2 = "Second string"; $str3 = "Third string"; $bytes = pack("C C A* C A* C A*", $count, length($str1), $str1, length($str2), $str2, length($str3), $str3); unpack("C", $count); $bytes = substr($bytes, 1); foreach $i (1 .. $count) { $length = unpack("C", $bytes); $str = unpack("xA$length", $bytes); # null byte skips length $bytes = substr($bytes, $length + 1); print "$str\n"; }
My question is - how can avoid the constant substr calls to remove the bytes I have already read?. I know I could use the "C/A*" template to read the length and string in one go, but that doesn't handle the variable number of strings.
In the real-life example, there may be millions of strings, and there may actually be other data types (ints) interspersed. I do actually read the binary file in chunks, so the actual number of bytes that substr is acting on is relatively small, but there must be a more efficient way? Can I have a pointer into the byte stream and call unpack on that? Maybe some of the modules that allow a file to be accessed as a scalar variable might help? Note: I did also try replacing the substr with a variant of the unpack, but it didn't seem to improve performance, and seemed to cause extra bytes to be consumed?: ($str, $bytes) = unpack("x C/A* A*), $bytes) Thanks!