Fun with bit strings :)
Here's compress and decompress subs with supporting hashes.
Using your test generator, I get about 55% compression most of the time.
This is better than gzip or bzip2 on the one test case I tried. gzip was below 30% and bzip2 was below 45%.
There are pathological cases however.
If you replace your rand(10) with 9 you will get a 675% expansion instead of compression.
The compressor maps 4 lines of your test input to 27 bytes and the decompressor does the opposite,
so I suggest you compress in multiples of four lines, and decompress in multiples of 27 bytes if you have to split up very large files.
decompress(compress($string)) does reproduce the input string exactly, however, no matter how many lines are in it (at least in my testing :)
#!/usr/bin/perl
# https://perlmonks.org/?node_id=1233613
use strict;
use warnings;
my @legal = grep !/11/ && tr/1// <= 3, glob '{0,1}' x 8;
my %code;
@code{@legal} = map { unpack 'b6', chr } 0 .. $#legal;
my %decode = reverse %code;
$_ = [ split ' ', '23456789' & tr/01/ ?/r ] for values %decode;
sub compress
{
my $coded = '';
for ( shift =~ /(.*)\n/g )
{
my @lookup = (0) x 123;
@lookup[ unpack 'C*', $_ ] = (1) x length;
for( my $group = 35; $group < 123; $group += 10 )
{
$coded .= $code{ join '', @lookup[$group .. $group + 7] };
}
}
return pack 'b*', $coded;
}
sub decompress
{
my $decoded = '';
for my $line ( unpack('b*', shift) =~ /.{54}/g )
{
my $decade = 33;
for my $group ( @decode{ unpack '(a6)*', $line } )
{
$decoded .= pack 'C*', map $decade + $_, @$group;
$decade += 10;
}
$decoded .= "\n";
}
return $decoded;
}
my $input = '';
for (1 .. 100)
{
for (my $x=0; $x<90; $x+=10)
{
my @c;
push(@c, int (rand(10)+$x));
push(@c, int (rand(10)+$x));
push(@c, int (rand(10)+$x));
push(@c, int (rand(10)+$x));
@c = sort{$a<=>$b}@c;
for (my $i = 1; $i < @c; $i++)
{
$input .= chr(33+$c[$i]) if $c[$i] != $c[$i-1] && $c[$i] != $c[$
+i-1]+1;
}
}
$input .= "\n";
}
#use Data::Dump 'dd'; dd $_ for $input =~ /.*\n/g;
print "\n input length ", length $input, "\n";
my $compressed = compress($input);
my $compressedlength = length $compressed;
print "compressed length $compressedlength\n";
my $restored = decompress($compressed);
if( $input eq $restored )
{
printf "\nMatched, compression ratio = %.1f%%\n",
100 * (1 - length($compressed) / length($restored));
}
else
{
print "----------------------------------------failed\n";
use Data::Dump 'dd'; dd $_ for $restored =~ /.*\n/g;
}
Output of typical run (100 random lines) :
input length 1507
compressed length 675
Matched, compression ratio = 55.2%
Thanks for the fun chance to play with long bit strings.