Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Out of Memory Error -- Possible Leak?

by Zaxo (Archbishop)
on Dec 13, 2005 at 23:44 UTC ( #516475=note: print w/replies, xml ) Need Help??


in reply to Out of Memory Error -- Possible Leak?

How much memory do you have? Your data structures are immense. It looks like you have perhaps 100,000,000 arrays in the hash, and even if most are not there you are autovivifying their keys ( scalar(@array) different anonymous hashes) and a reference to an empty array.

All those structures cost fifty-some bytes each plus a hefty multiple of the number of data elements.

I don't see a leak, just prolific use of memory. Can you read and use less of the data at a time?

After Compline,
Zaxo

  • Comment on Re: Out of Memory Error -- Possible Leak?

Replies are listed 'Best First'.
Re^2: Out of Memory Error -- Possible Leak?
by Anonymous Monk on Dec 13, 2005 at 23:56 UTC

    The machine itself has 32 GB, but I'm using a 32-bit compiled Perl so presumably it can only address 4 of those.

    Also, 100 million arrays is a realistic estimate (its not sparse). I had thought that by using DBM::Deep I would be storing the majority of this data on disk, so that the actual amount stored in memory would be much smaller than 100 million x size(data-structure). Am I misguided on that?

Re^2: Out of Memory Error -- Possible Leak?
by emilper (Novice) on Dec 14, 2005 at 11:59 UTC
    maybe it's because the garbage collector does not have time to run ...
      That isn't possible, because Perl doesn't use a separate garbage collector. It simply reference-counts all its data (scalars, arrays, hashes etc) and frees an item when its reference count drops to zero.

      (This is why circular data structures cause memory leaks.)

        then how do you explain this ?

        try:

        perl -e ' use Tk; while (1) { my $top = new MainWindow; }'
        

        it will fill your RAM at a rate of about 1MB per second ...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://516475]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2022-12-09 23:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?