in reply to Processing ~1 Trillion records

Why are you bothering (and paying the transport & memory costs) to query 5 fields for every record, when you know that 3 of those fields are always going to be '9', 'STR', & 'STR1' respectively?

Why build a 2-level hash:

$data{ $dat[0] }{ $dat[1] } = $tmarker;

when the first level is always '9'.

Ditto this:

$lines{ $dat[0] } = "";

And why are you doing this:

if( !defined $dat[3] ) { $dat[3] = 0; } if( !defined $dat[4] ) { $dat[4] = 0; }

When the query dictates that fields 3 & 4 can never be undefined.

What is the point of this:

$datachr{ $dat[1] }{ $dat[3] } = $dat[4];

when $dat[3] & $dat[4] are constants?

Ditto this:

$chr{ $dat[3] } = "";

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong