http://qs321.pair.com?node_id=942943


in reply to Re^2: Efficient way to handle huge number of records?
in thread Efficient way to handle huge number of records?

An alternative to building a hash in memory would be to have a disc based hash like DBM::Deep. Size is limited by the size of your disc, it is fast and it is particularly suited to “write seldom, read many” which appears to be your case.

However, it seems from your description that the “unknown query” would parse/search on the value of each key (“search through a hash”) i.e. look at each value in turn for each query. I always think of a hash as a “lookup table” and it doesn't look like this is what you'll be doing. It maybe that an array would be more suitable, less overhead at least. Something like NAME1|SEQUENCE1. According to the docs D::D can build disc based arrays too although I have never tried.

How unknown are your unknowns? Could there be general subsets/groups that could be preprocessed; you say there is additional processing to be done on the results – could that be done upfront? A series of additional lookups that could identify a smaller group of records to search – divide and conquer? If you did choose the RDBMS route are there indices that could be built that would help find what you're after. If there aren't it would, imo, likely negate any advantage.

If you have to process each record for each search then whatever method you adopt is going to be slow and possibly not suitable for a website.

If you could up come with an outline of any known unknowns there is a good chance the monks could give you some pointers on how to avoid what seems like a brute force approach. It would be a challenge many of them would be unable to resist. :-)

Replies are listed 'Best First'.
Re^4: Efficient way to handle huge number of records?
by jethro (Monsignor) on Dec 11, 2011 at 14:45 UTC

    I think you are reading too much into his use of the word "search", but you have a point there. If he ever wants to do lots of searches for parts of a key (i.e. "contains" instead of "equals"), a hash will be as bad as a simple array.