Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have been running a perl prog for the log files. though there is 4Gb of memory, the program is unable to process the data completely and is terminting well before, citing Out of Memory. I wouldl ike to know the reason. I wld also like to know if there is a better way to handle large amt of data other than using DBM

Replies are listed 'Best First'.
Re: Out of memory
by Zaxo (Archbishop) on Oct 30, 2002 at 05:08 UTC

    Are you slurping the whole file? Producing an array of all the records? Please show some code. Guessing games are fun, but they won't help you much.

    After Compline,
    Zaxo

[reply]
      I have used hashes in the matrix form.ie nested hashes. The code stopped executing when the memory was arnd 3Gb. The memory of the system is 4Gb.
[reply]
Re: Out of memory
by strat (Canon) on Oct 30, 2002 at 10:51 UTC
    Sounds like memory fragmentation...

    If you use a perl version >= 5.6, try using the 5.005_03 instead. Especially for more complex hash-structures, it may use less memory, so you can create a bigger structure.

    But this is just a quick hack, not a real solution; a better way may be to split up the data and try to keep only little chunks in memory. Or use a 64bit operating system... :-)

    Best regards,
    perl -e "s>>*F>e=>y)\*martinF)stronat)=>print,print v8.8.8.32.11.32"

[reply]

Back to Seekers of Perl Wisdom