http://qs321.pair.com?node_id=944887


in reply to Re: Displaying/buffering huge text files
in thread Displaying/buffering huge text files

And 8 bytes for each offset? ... 5 byte offset: 1 Tb;

I, for one, would like to see your code for building an 5-byte index?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?
  • Comment on Re^2: Displaying/buffering huge text files

Replies are listed 'Best First'.
Re^3: Displaying/buffering huge text files
by JavaFan (Canon) on Dec 23, 2011 at 10:32 UTC
    I, for one, would like to see your code for building an 5-byte index?
    Storing a million 5-byte integers using 5 million bytes for data, and 44 bytes additional overhead: 
    my $buffer = "";
my $BYTES = 5;
my $BITS_IN_BYTE = 8;
my $FULL_BYTE = (1 << $BITS_IN_BYTE) - 1; 

sub store {
    my ($pos, $value) = @_;
    for (0 .. $BYTES - 1) {
        vec($buffer, $pos * $BYTES + $_, $BITS_IN_BYTE) =
           ($value >> ($BITS_IN_BYTE * ($BYTES - 1 - $_))) & $FULL_BYT
+E;
    }
}

sub fetch {
    my ($pos) = @_;
    my $sum = 0;
    for (0 .. $BYTES - 1) {
        $sum <<= $BITS_IN_BYTE;
        $sum += vec($buffer, $pos * $BYTES + $_, $BITS_IN_BYTE);
    }
    $sum;
}

#
# Testing
#
my $TEST_SIZE = 1_000_000;
my @offsets = map {int rand 1 << ($BYTES * $BITS_IN_BYTE)} 1 .. $TEST_
+SIZE;

for (my $i = 0; $i < @offsets; $i++) { 
    store $i, $offsets[$i];
}
for (my $i = 0; $i < @offsets; $i++) { 
    my $val = fetch $i;
    die unless $val == $offsets[$i];
}

use Devel::Size 'size';
use 5.010;

say "Index size: ", size $buffer;
__END__
Index size: 5000044

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom