The stupid question is the question not asked | |
PerlMonks |
comment on |
( [id://3333]=superdoc: print w/replies, xml ) | Need Help?? |
Let's take a history-lesson from ... punched cards, of all things. When you have a large amount of data, sort it. If you have multiple data streams to integrate, sort each one by the same key. When you do this, all the records having any key value will be adjacent, and within any gaps there are known to be no keys at all. No searching is required, and you never need to consider anything more than: this record, and the preceding one. The thing that is eating your lunch here is a familiar one: thousands (or millions) of repetitive searches. “Come with me back to the days when more data than this was processed using much less computer power than today you will find in a dishwasher. It was not a world of hard drives: it was a world of magnetic tapes.” The procedures that were adopted out of necessity back then, are just as efficient today. The payoff for your efforts can be ... stunning. In reply to Re: speeding up row by row lookup in a large db
by sundialsvc4
|
|