Before showing you the code, some notes.
- You are mixing text (the boundary delimiters) and binary data. This is bad
- You are writing out hexadecimal representations of numbers, but you want to save space. This makes little sense
- You should really use pack
Now, I changed the encoding. Now it is:
- one long, for the index length
- the index. Each entry in the index is made of:
- one long: the index number
- one long: the string length
- some bytes: the string itself
- the content, as a series of longs
And now, the code:
Encoder:
#!/usr/bin/perl
@lines = <>;
binmode STDOUT;
$x = 1;
foreach $ln (@lines) {
@words = split /\s+/, $ln;
foreach $word (@words) {
$index{$word} = $x++;
$count{$word}++;
}
}
$idx='';
foreach $key (keys %index) {
$idx.=pack("N(N/a*)",$index{$key},$key);
}
print STDOUT pack("N",length($idx)),$idx;
foreach $ln (@lines) {
@words = split /\s+/, $ln;
foreach $word (@words) {
print STDOUT pack("N",$index{$word});
}
}
Decoder:
#!/usr/bin/perl
binmode STDIN;
undef $/;
$i=<>;
$is=unpack "N",$i;
$ind=substr($i,4,$is);$con=substr($i,4+$is);
%index=unpack "(N(N/a))*",$ind;
@con=unpack "N*",$con;
print join(' ',@index{@con});
Note the use of binmode and the unsetting of $/ (aka input record separator).
--
dakkar - Mobilis in mobile