Out of Memory

instinct_4ever has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Out of Memory by ikegami (Patriarch) on Oct 25, 2009 at 04:39 UTC
Use a trie instead of a hash table. For future reference, `if(exists $wordCount{$_}) { $wordCount{$_} = $wordCount{$_} + 1; } else { $wordCount{$_} = 1; }` [download] can be shortened to `++$wordCount{$_};` [download] It won't help your memory problem, just your code clarity.	[reply] [d/l] [select]
Re^2: Out of Memory by Anonymous Monk on Oct 25, 2009 at 05:18 UTC
Tree::Trie?	[reply]
Re^3: Out of Memory by ikegami (Patriarch) on Oct 25, 2009 at 05:39 UTC
That is indeed an implementation of the previously mentioned data structure. I've never used it or other modules, thus the lack of recommendation.	[reply]
Re: Out of Memory by virtualsue (Vicar) on Oct 25, 2009 at 10:57 UTC
Outside of a redesign (probably the best choice IMO), you can have a look at http://www.ohloh.net/p/Sack-perl. dpavlin has been working on a way to shard hashes across several machines - Google like scalability using perl?.	[reply]
Re: Out of Memory by afoken (Chancellor) on Oct 25, 2009 at 09:02 UTC
Use an RDBMS, e.g. SQLite or PostgreSQL. Stuff the lines read into a table, create an index on the column, and use something like `SELECT line,COUNT(*) FROM table GROUP BY line` to fetch the counts. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l]
Re: Out of Memory by JavaFan (Canon) on Oct 25, 2009 at 10:40 UTC
Without knowing why you are putting so many keys in a hash, it's hard to say what to do. (Note that it matters less how many lines there are in the file, as well as the number of different words). One obvious savings you can do is chopping off the newline - that would save you a couple of Mb. But you might consider using a disk bound datastructure. Perhaps a database, or one of the DB files. A trie was suggested as well, but I'm not sure how much it will save. Obviously, the amount of string data is reduced, but at the cost of introducing more hashes (or arrays), which themselves come with quite a lot of memory overhead. It will depend on the prefix duplication in the data set.	[reply]
Re: Out of Memory by BrowserUk (Patriarch) on Oct 25, 2009 at 16:07 UTC
If all you're doing is counting the unique lines, sorting the file first and then counting the consecutively similar lines will require very little memory and probably be quicker than loading the data into a db. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP PCW It is as I've been saying!(Audio until 20090817)	[reply]
Re: Out of Memory by Jenda (Abbot) on Oct 26, 2009 at 11:17 UTC
use BerkeleyDB; Jenda Enoch was right! Enjoy the last years of Rome.	[reply]
[DUP] Re: Out of Memory by ikegami (Patriarch) on Oct 25, 2009 at 04:41 UTC
[ Duplicate post. Please ignore ] Use a trie instead of a hash table. For future reference, `if(exists $wordCount{$_}) { $wordCount{$_} = $wordCount{$_} + 1; } else { $wordCount{$_} = 1; }` [download] can be shortened to `++$wordCount{$_};` [download] It won't help your memory problem, just your code clarity.	[reply] [d/l] [select]