in reply to Re^2: Memory utilization and hashes
in thread Memory utilization and hashes

Right, so it is much more complicated in my real code. I create an array for the multiple answers as such and am doing some funky checks to print out the info because the index number can be reused. So, say index 2 has an answer provided, then 2 can be re-used in another query. I then dump what is left of the has at the end of the code for those items that did not get re-used and replaced.

Like I said, it is really messy in "real life".

I will provide example code that is closer to my real code shortly but my real question is, I supposed, if a hash is the right way to do this after all due to memory issues and such.

Replies are listed 'Best First'.
Re^4: Memory utilization and hashes
by bfdi533 (Friar) on Jan 17, 2018 at 22:16 UTC

    I did try to use Devel::Size to see if the memory actually goes down so am writing the size of the has to a log file every time I "dump" a line and the size never decreases since I have been testing it.

    The here is an example. First column is line count into the file being processed, the second is the index (equivalent to $vals1) and the last the size of the %pairs hash. Here the size is 122MB for the %pairs hash ... 

    ...
424872: e5c651161 (122480629)
424875: 6d6148148 (122481928)
424886: 108038067 (122484667)
424890: 4db238067 (122487257)
424892: 502c57487 (122488556)
424895: c53c57539 (122489855)
424896: 578757487 (122489855)
424923: 300959147 (122495178)
424928: a9bb41168 (122496165)
424936: dfc243245 (122499555)
424937: 0a9534098 (122499555)
424944: 666b34098 (122501654)
424954: 494949982 (122504073)
424956: 182939296 (122505372)
424960: c1ad46207 (122507962)
424962: 3d1249982 (122507962)
424968: 3c1336561 (122512355)
424974: b24939296 (122514993)
424987: 3c7b36561 (122517700)
424998: eb1544993 (122520311)
425005: 818a49369 (122521727)
...

    [download]
[reply]
[d/l]
      Memory allocation is a fairly complicated subject, and I'm not sure what Devel::Size does exactly. Moreover, I am really not a specialist on these things, so what I will say next is not at all from any insider knowledge on how Perl (and the underlying C functions) allocate memory, but from what I have read from people knowing these things presumably much better than I do. So, take it with a pinch of salt, I may be misunderstanding some of these things.

      In general, I would not expect the overall memory footprint to decrease when you delete some data structures. Even if some memory has been freed, it will probably not show in these stats (due, among other things, to memory fragmentation). But if you have freed some of it by deleting some hash entries, then there is good chance that your program will be able to reuse the freed memory, so that creating a new hash entry will not increase the overall memory usage.

      Update: Fixed wrong link for the Devel::Size module.

[reply]

        Noted; thanks. The memory re-use does make sense.

[reply]

In Section Seekers of Perl Wisdom