http://qs321.pair.com?node_id=938688

An array to hold 1000 integers requires 32k:

@a = 1 .. 999;; print total_size \@a;; 32144

A hash to hold 1000 keys & integer values requires 100k:

\$h{ \$_ } = \$_ for 1.. 999;; print total_size \%h;; 109055

So, on a machine with less memory than some musical birthday cards, the hash or array one-liners will perform this task efficiently, and scale linearly for files containing at least 18,446,744,073,709,551,616 lines.

To put that into perspective, it represents a single data file containing 16 Exabytes. Or approximately a million times more data than Google's entire storage capacity. If the OP can afford the amount of disk required to hold the file, it seems very unlikely that he'll have any trouble affording the 100k of ram.

In the meantime, it would take a computer running at 10Ghz and able to perform 1 comparison per clock cycle, 3,741 years to sort that file, assuming no other time costs including IO or memory.

So do the world a favour, and apply a little, the merest modicum, of thought to the problem at hand, before trotting out your olde worlde compooter wisdoms. Regurgitating received knowledge, long since superseded, as a substitute for actually thinking about the problem, does no one any good.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^3: "Just use a hash": An overworked mantra?
by sundialsvc4 (Abbot) on Nov 19, 2011 at 17:38 UTC

For this problem, in which the known solution-space is constrained to what can fit into a reasonably sized hash and in which the total number of records and data-streams also fits into memory ... a memory-based solution works just fine, and there is utterly no reason to trundle out n-digit numbers to “prove” your point.

My original comment, which I said even at that time was ancillary to the original discussion, is that there do exist other classes of problems which for various reasons do not lend themselves well to the “random-access based” (and to “memory-based”) approaches that might occur to you on first-blush.   This might not be one of those cases, but it does not invalidate the fact that such problems do exist.   In those problems, the incremental costs of virtual-memory activity become a death by a thousand cuts.   A fundamental change of approach in those cases transforms a process that runs for days, into one that runs in just a few hours.   I have seen it.   I have done it.   “Batch windows” are a reality for certain common business computing jobs.   Last year I worked on a system that processes more than a terabyte of new data, assimilated from hundreds of switching stations, every single day, and this was the change that gave them their system back.

I was really, really hoping that in this case you wouldn’t rush out once again to prove how smart you are.   Let alone, as so many times before, publicly and at my expense.   Enough.

For this problem, in which the known solution-space is constrained to what can fit into a reasonably sized hash and in which the total number of records and data-streams also fits into memory ... a memory-based solution works just fine,

Ignoring the silly bit about "records and data-streams" fitting in memory. Exactly!

My mother had a brilliant solution to the problem of grease stains on carpets that involved brown paper and an iron; but you don't see me trotting out here at random.

there is utterly no reason to trundle out n-digit numbers to “prove” your point.

Beg to differ. There was a reason.

Your continued insistence to trot out the description of something that might prove to be a suitable to solution to some other problem at some other place and time gave me that reason.

I was really, really hoping that in this case you wouldn’t rush out once again to prove how smart you are. Let alone, as so many times before, publicly and at my expense.

The only smarts involved, is your exhibited lack thereof in posting inappropriate solutions to questions.

The only expense involved, is the time wasted by recipients of your "wisdoms", as they chase down blind alleys following them.

Enough.

We found something we can agree on.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.