Re^4: Memory utilization and hashes

I did try to use Devel::Size to see if the memory actually goes down so am writing the size of the has to a log file every time I "dump" a line and the size never decreases since I have been testing it.

The here is an example. First column is line count into the file being processed, the second is the index (equivalent to $vals1) and the last the size of the %pairs hash. Here the size is 122MB for the %pairs hash ...

...
424872: e5c651161 (122480629)
424875: 6d6148148 (122481928)
424886: 108038067 (122484667)
424890: 4db238067 (122487257)
424892: 502c57487 (122488556)
424895: c53c57539 (122489855)
424896: 578757487 (122489855)
424923: 300959147 (122495178)
424928: a9bb41168 (122496165)
424936: dfc243245 (122499555)
424937: 0a9534098 (122499555)
424944: 666b34098 (122501654)
424954: 494949982 (122504073)
424956: 182939296 (122505372)
424960: c1ad46207 (122507962)
424962: 3d1249982 (122507962)
424968: 3c1336561 (122512355)
424974: b24939296 (122514993)
424987: 3c7b36561 (122517700)
424998: eb1544993 (122520311)
425005: 818a49369 (122521727)
...
[download]

Comment on Re^4: Memory utilization and hashes Download Code

Replies are listed 'Best First'.
Re^5: Memory utilization and hashes by Laurent_R (Canon) on Jan 17, 2018 at 22:40 UTC
Memory allocation is a fairly complicated subject, and I'm not sure what Devel::Size does exactly. Moreover, I am really not a specialist on these things, so what I will say next is not at all from any insider knowledge on how Perl (and the underlying C functions) allocate memory, but from what I have read from people knowing these things presumably much better than I do. So, take it with a pinch of salt, I may be misunderstanding some of these things. In general, I would not expect the overall memory footprint to decrease when you delete some data structures. Even if some memory has been freed, it will probably not show in these stats (due, among other things, to memory fragmentation). But if you have freed some of it by deleting some hash entries, then there is good chance that your program will be able to reuse the freed memory, so that creating a new hash entry will not increase the overall memory usage. Update: Fixed wrong link for the Devel::Size module.	[reply]
Re^6: Memory utilization and hashes by bfdi533 (Friar) on Jan 17, 2018 at 22:47 UTC
Noted; thanks. The memory re-use does make sense.	[reply]


laziness, impatience, and hubris
	PerlMonks