Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re: When is a flat file DB not enough?

by Nitsuj (Hermit)
on Jul 05, 2001 at 05:35 UTC ( #93987=note: print w/replies, xml ) Need Help??

in reply to When is a flat file DB not enough?

Berkley DB has been mentioned a few times here. I think that this is probably the best idea in your situation (though I would strongly encourage migrating to a good sturdy DBMS (if you are sorting that much data). For just a couple hundred records, you will probably not notice much difference, but if this sucker is getting big... Well...

Of course, you should use a database hash and that will sort things out nicely

Another thought, is that you can breathe a LOT of life into flatfiles with a few simple methods.
One simple one is to use your filesystem to provide some of the services of a DBMS, but I don't really recommend that (IE, directories labelled as user numbers)
Another idea is to put some forethought into your file format. You can put tables in the front to sort the data, run simple hashes over files to speed up searching, use tree implementations and such. An important fact is that you don't actually need to visit each record in order to search a file. If the data is ordered, you can order records, and search positionally (kind of like the number game where the computer says "higher" or "lower"). This will speed up your search time exponentially (literally). Search time for a B-Tree is log(2)n visitations, through the judicious use of file pointers, this can be used in any which way you like. The simple fact of the matter though, is that implementing such a system is rather mind racking (there is probably a pm that allows something to this effect).

I personally would try splitting your file across some criteria into smaller files, which can be searched flatly, and hand it off to IT for a proper DBMS if access time gets to be a problem.

Just Another Perl Backpacker

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://93987]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2022-05-29 11:34 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (101 votes). Check out past polls.