http://qs321.pair.com?node_id=662118


in reply to why won't the hash go away?

See "How can I free an array or hash so my program shrinks?" in perlfaq3 ... Just clearing/undef'ing a variable won't release the memory back to the system (though it is then available to the program again).

Replies are listed 'Best First'.
Re^2: why won't the hash go away?
by WoodyWeaver (Monk) on Jan 12, 2008 at 23:36 UTC
    Ok, fair enough. Its disappointing that I can't hand it back to the system, but I don't really need that (although it would be nice, of course.)

    What I want to do is to shuffle the cards and start over. I don't really mind keeping the memory around, really. I just want to make the hash list go away so I can start a new one.

    The program runs for a couple of hours, then gives up and decides to go run this re-init routine, then spends MORE THAN TEN HOURS just trying to execute %positions = ().

    I guess what you are telling me is that once it starts to thrash, I'm toast, and that page management or whatever just isn't going to help me. So even my approach to break the hash table into smaller pieces isn't going to help.

    Pity. Sounds like the solution will be to follow the re-exec approach; don't try to re-init, just die and restart yourself. Hm. Not sure I like that -- granted, I'm anthromophising shamelessly, but it then boils down to "play solitaire and win, or die"...

    --woody

      You could set it up so that each shuffle is done after a fork, and just wait for that one to end/die before going to the next. Something like (untested; off top of head & docs, but i think is the general outline):
      my @deck = create_deck(); while(1){ # or whatever; loop over specific deals, or do just N deals +, etc. my $pid = fork; die unless defined $pid; if($pid){ # parent wait; # report on $? if desired next; } do_heavy_lifting(\@deck); exit; # w/an error code based on result if desired } sub do_heavy_lifting { ... my %positions = ....; ... }
      So the %positions isn't created/populated (made huge) until in the fork'd process, so when that exits, the memory will get freed to the system, but your program is still going from the parent process.

        Memory management via fork is a good idea; I've used it in several situations (usually when I'm going to leak memory).

        The problem WoodyWeaver may have is that when the child exits normally, it will still go through the ten hours of data structure destruction, and the parent will wait the whole time. Taking some code from davidrw and some from my own node,

        use Set::Light; my $pid = fork; die "Can't fork: $!" if ! defined $pid; if($pid){ # parent my $start_time = time; wait; printf "Child exited after %d seconds\n", time - $start_time; } else { # child my $start_time = time; my $set = Set::Light->new(); for( my $i = 0; $i < 10_000_000; $i++ ) { $set->insert( $i ); } printf "Child done after %d seconds\n", time - $start_time; exit; } __END__ Child done after 36 seconds Child exited after 163 seconds

        The solution to this is not to let the child exit normally (i.e., kill it). In the above code, if I change "exit" to "kill 9 => $$", the OS does the memory reclamation, and the output looks like this:

        Child done after 32 seconds Child exited after 32 seconds
        Its a good plan. I tend to prefer a watchdog approach -- a second program that periodically wakes up, scans the process table, sees if enough copies of (whatever) are running, and if not launch enough additional to bring it up to snuff. But in any event, it sounds like you are suggesting re-exec as the best approach.