Re^3: a large text file into hash

I tried to create the hash and then tie it,

Tie::IxHash a) doesn't store to disk; b) use 2 or 3 time as much memory as a standard hash. It's purpose is to remember the order in which the keys of the hash were added which is unnecessary for your use. You should not be using this module.

If you are going the tie'd hash root, then you need to use a module that ties the hash to a disk file. Previously I'd have recommended BerkeleyDB, but since Oracle grabbed Sun, you have to sign up and agree to let them do whatever they want before they'll let you download anything.

There are alternatives but I don't have much experience of them, so I cannot make a recommendation.

But, if you have 50GB of ram available, then you ought to be able to hash your 1 GB file in memory with ease.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^3: a large text file into hash

Replies are listed 'Best First'.
Re^4: a large text file into hash by perl_lover_always (Acolyte) on Jan 28, 2011 at 14:12 UTC
well, since I make the hash once and I want to use it several times later, then I'd prefer to keep it in the hard disk for the later access. what is your suggestion in this scenario?	[reply]
Re^5: a large text file into hash by BrowserUk (Patriarch) on Jan 28, 2011 at 14:48 UTC
As I said, BerkeleyDB is the most recommended solution for disk-based hashes. Alternatives include MLDBM, SQLite_file, DBM::Deep etc. Or you could use an RDBMS (Postgres/MySQL/other). Perhaps via Tie::DBI. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]


Welcome to the Monastery
	PerlMonks