|
Re: Why I got nearly out of memory, and never recover from that?
by moritz (Cardinal) on Jun 16, 2009 at 06:48 UTC
|
How does Perl manage memory? Does an object be automatically released, if it is only refered as a local variable within a block?
Yes, it's released, but not given back to the operating system. Which means that the memory can be used for other Perl variables, but not from other processes.
Another question: In loop conditions, after each time of run, does the memory of local variables -- which are inited within the loop BODY(not the heading of loop) -- being deleted and released, and recreated next time?
Yes. (Although there's an optimization which takes care that only the contents of the variable are deleted, not the variable itself).
| [reply] |
|
|
I think the Op has a "memory leak" for some reason. There are a bunch of ways that this could happen. But this post is right, Perl will re-use memory if it can. It NEVER gives memory back to the O/S and there are many ways that Perl memory gets "re-cycled" within Perl.
A "long lived" Perl program will grow to a maximum memory "footprint" and THEN that size remains constant. If you see that memory is always increasing (and not "leveling off"), then one of your objects is not releasing its memory for Perl to recycle. The main point here is that a Perl program will reach a max memory size and that's it!
| [reply] |
|
|
To all above:
The most direct problem it caused is: it seems using hard disk prior than "true" memory, which caused the program runs rather slow. How could I improve it?
| [reply] |
|
|
|
|
Re: Why I got nearly out of memory, and never recover from that?
by dsheroh (Monsignor) on Jun 16, 2009 at 12:22 UTC
|
You haven't said what kind of processing you're doing, but, in the vast majority of cases, you don't actually need to load the whole file into memory. If the files have any kind of internal structure, you can probably read them in corresponding chunks - whether that's line-by-line or record-by-record - to do your processing with only one chunk (or maybe a small number of chunks, depending on the nature of your processing) in memory at a time.
If your processing amounts to "read in the whole file, then loop over pieces of the file to handle each piece in turn", then you almost certainly do not need to read it all in at once. | [reply] |
Re: Why I got nearly out of memory, and never recover from that?
by llancet (Friar) on Jun 16, 2009 at 08:02 UTC
|
Thanks to everybody above! And I think I've got some idea to overwhelm that:
Maybe I can wrote the program into two: one for iterate through all files, the other do the "true" process on each file. After finished processing each file(maybe very large), the second process is completed, resource is released, and my computer won't get filled. | [reply] |
|
|
I don't know what you are doing. But a virtual memory space of 2 GB is reasonable for 32 bit Win XP or *nix. You can have 4 processes that each use 1.5 GB of memory, with only total of 0.5 GB of physical memory. It will run..It will run like a "herd of turtles", but it will run.
| [reply] |