I don't see what garbage collection has to do with this. The strings in question don't have any references to them, so the reference counter shouldn't have any problem knowing that they're not in use.

That means it either has to keep it there always, or free it always (or do some kind of heuristic, which should usually mean keep it, since allocating memory is expensive, and if you're using a large string now, chances are, you'll be using a large string again some time soon).

What perl currently cannot do, is free "old, unused" scalars when it's running out of memory. It has to decide when the scalar is going out of scope. allocating and freeing each scalar every time that happens would probably slow down the interpreter a lot.

That means it either has to keep it there always, or free it always (or do some kind of heuristic, which should usually mean keep it, since allocating memory is expensive, and if you're using a large string now, chances are, you'll be using a large string again some time soon).

Especially for a large string I wonder in how far the gain of avoiding the deallocation is considerable with regard to filling the buffer with the string and working with it.

Of course you're right: I shouldn't copy 500MB strings around too much, however I chose such a drastic length to make the effect clear. Even if the strings were smaller, I'd say I could use 1MB of memory for better things than storing 32 long-forgotten scalars of 32kB each (or even smaller ones). I am not really good at making up realistic scenarios, but I'm interested to know: Would you have anticipated perl's behaviour if you had just seen my code samples above?

I'm glad that kyle seems to agree that it would be nice if perl dealt better with these unused scalars. Besides, it obviously doesn't reuse the buffer if a subroutine calls itself recursively... but I haven't tested the memory consumption for this case yet.

Oh, I think I see what you're saying now. When the last reference goes out of scope is the only time it gets to make a decision about whether to deallocate the memory used for the variable. We don't necessarily know then whether the variable will be used again or not, or what for, so it's not a very good time to make that decision.

I'm not sure I'm convinced that deallocating would be a bad thing. Obviously, it depends on what the program is doing. I'd be tempted to take some kind of heuristic approach, but even then I'd want to do some testing to find where the cost/benefits are.

But it wouldn't be terribly hard to implement some improvements here.

Perl could keep a LRU of still-allocated but unused "large" SVs and periodically free ones that haven't been re-used in the last period of time.

Note that what might look like the easiest "fix", freeing SVs for lexical variables if they are above a certain size, could have serious draw-backs, at least on some systems. Having done something like this on Win32, this can be a great way for heap fragmentation to cause your process to run out of virtual memory. Perhaps most other systems have smarter malloc()s and so aren't susceptible, but I'm not certain of that.

It also might be tricky to pick the proper parameters for what consistutes "large" and what the right minimum duration should be before the large allocation is declared "unlikely to be re-used" and free()d.

- tye        

The strings in question don't have any references to them

Not true. The pad that refers to them when the function is being executed still refers to them when the function isn't being executed.

It could be changed to be true, so this nit pick is not relevant to the conversation.