Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:
I am working on a script to read and interpret a binary file, which consists of a variable number of variable length strings.
There are count values embedded in the file for the number of strings, and the length of each string, e.g (artificial example):
In the real-life example, there may be millions of strings, and there may actually be other data types (ints) interspersed. I do actually read the binary file in chunks, so the actual number of bytes that substr is acting on is relatively small, but there must be a more efficient way? Can I have a pointer into the byte stream and call unpack on that? Maybe some of the modules that allow a file to be accessed as a scalar variable might help? Note: I did also try replacing the substr with a variant of the unpack, but it didn't seem to improve performance, and seemed to cause extra bytes to be consumed?: ($str, $bytes) = unpack("x C/A* A*), $bytes) Thanks!
There are count values embedded in the file for the number of strings, and the length of each string, e.g (artificial example):
The script itself contains something like this:NumberOfStrings (1 byte) Str1Length (1 byte) Str1Characters (Str1Length bytes) Str2Length (1 byte) Str2Characters (Str2Length bytes) ... StrNLength (1 byte) StrNCharacters (StrNLength bytes)
My question is - how can avoid the constant substr calls to remove the bytes I have already read?. I know I could use the "C/A*" template to read the length and string in one go, but that doesn't handle the variable number of strings.# Create an example byte-sequence $count = 3; $str1 = "First string"; $str2 = "Second string"; $str3 = "Third string"; $bytes = pack("C C A* C A* C A*", $count, length($str1), $str1, length($str2), $str2, length($str3), $str3); unpack("C", $count); $bytes = substr($bytes, 1); foreach $i (1 .. $count) { $length = unpack("C", $bytes); $str = unpack("xA$length", $bytes); # null byte skips length $bytes = substr($bytes, $length + 1); print "$str\n"; }
In the real-life example, there may be millions of strings, and there may actually be other data types (ints) interspersed. I do actually read the binary file in chunks, so the actual number of bytes that substr is acting on is relatively small, but there must be a more efficient way? Can I have a pointer into the byte stream and call unpack on that? Maybe some of the modules that allow a file to be accessed as a scalar variable might help? Note: I did also try replacing the substr with a variant of the unpack, but it didn't seem to improve performance, and seemed to cause extra bytes to be consumed?: ($str, $bytes) = unpack("x C/A* A*), $bytes) Thanks!
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Faster way to parse binary stream
by BrowserUk (Patriarch) on Jun 19, 2007 at 19:50 UTC | |
by ikegami (Patriarch) on Jun 19, 2007 at 20:29 UTC | |
by BrowserUk (Patriarch) on Jun 19, 2007 at 21:29 UTC | |
by Roy Johnson (Monsignor) on Jun 19, 2007 at 20:29 UTC | |
by Anonymous Monk on Jun 19, 2007 at 21:27 UTC | |
Re: Faster way to parse binary stream
by ikegami (Patriarch) on Jun 19, 2007 at 19:50 UTC |
Back to
Seekers of Perl Wisdom