Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re^4: RFC: Abusing "virtual" memory

by sundialsvc4 (Abbot)
on Nov 28, 2007 at 04:46 UTC ( #653479=note: print w/replies, xml ) Need Help??


in reply to Re^3: RFC: Abusing "virtual" memory
in thread RFC: Abusing "virtual" memory

“Why sort your keys externally when you can get the DB to do it for you?”

Indeed...

To seek to answer that question, I give you today's workload of 11,344,209 telephone call records. You have exactly 4 hours wall-time to process them. If you imagine that you have enough time to put all those records into a B-tree-indexed random file, I have a bridge to sell you. Instead, to solve this problem and to do so consistently on a daily basis, it will be necessary for you to accomplish the same workload -- with utter reliability and consistency -- much faster.

It may come as an utter and complete shock to you to fathom that your grandfathers, armed with nothing more punched-card tabulators and sorters, with nary a digital computer in sight, could do that. Every day. Under wartime conditions.

Replies are listed 'Best First'.
Re^5: RFC: Abusing "virtual" memory
by snowhare (Friar) on Dec 09, 2007 at 03:37 UTC
    #!/usr/bin/perl use strict; use warnings; use BerkeleyDB; my $db_file = '/home/snowhare/perl_monks/example.db'; unlink $db_file; my $db = BerkeleyDB::Btree->new( -Filename => $db_file, -Cachesize => 700_000_000, -Flags => DB_CREATE, ); srand; for(my $count = 0; $count < 12_000_000; $count++) { my $random_value = rand(16776216); my $status = $db->db_put( "$random_value" => "$count" ); } undef $db; [snowhare@blue-bay perl_monks]$ time ./big_btree.pl real 3m47.121s user 3m7.978s sys 0m4.884s
    This was on a desktop class machine with 1.5GBytes of RAM and an AMD Athlon 64 3000+ processor running Fedora Core 6 Linux.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://653479]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2021-01-22 00:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?