Is there any way you can post sample code that demonstrates the issue and indicate how many files are being manipulated?

We need a little more information than "I have a problem with large hashes. How do I solve it?" if you want a better answer than "Use a tied hash" or "Install more memory".

Generally speaking, when I've had that problem, it was that there was a memory limit via ulimit - removing that limit 'solved' the problem. At least, insofaras letting me use way more memory. Perhaps you have an underlying memory problem - wasting memory or leaking it, we can't be sure from your description. Assuming that not to be the case, though, it's probably a ulimit on memory.

"Doctor, it hurts when I do this!"

"Then don't do that."

So the answer is simple - don't write perl programs that consume more memory than your system can allocate.

s**lil*;  $*=join'',sort split q**;  s;.*;grr; &&s+(.(.)).+$2$1+; $; =
qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/)
[download]

1. Step through the program with the debugger and check the memory size to see where you are in the code when it increases:

$ perl -de0

Loading DB routines from perl5db.pl version 1.28
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):   0
  DB<1> !!ps v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
...
 4763 pts/1    S+     0:00      0  1013  4874  4348  0.4 perl -de0
  DB<2> @array=(1..1_000_000)
  DB<3> !!ps v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
...
 4763 pts/1    R+     0:01      0  1013 103142 102400  9.8 perl -de0
[download]

or,

2. Take a look at "Debugging Perl memory usage" in perldebguts and see if using PERL_DEBUG_MSTATS and Devel::Peek::mstat would help.

...but first look at the output of perl -V and see if it says "usemymalloc=n", in which case you won't have any memory statistics... :-(

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3959
virtual memory          (kbytes, -v) unlimited
file locks                  (-x) unlimited
[download]

Look into the method you're using to process the files. For example, do you really need to store the entire file into the hash? From your description, it seems you may be doing that. Alternatively, if you need the contents of all the files, you could try tying the hash to a database.

Without some more information about what you are trying to accomplish ("read some files and storing those lines into a hash" is not very descriptive), it will be impossible to give anything other than quite generic advice.