Re^4: Managing System Memory Resources

Effectively, that is exactly what the OP's test script is doing. Except that the data is being generated internally, 300 bytes at a time, rather than being read from an external source.

There is no reason that I am aware of why either version of the OP's testcase should be consuming large amounts of memory. It certainly doesn't on my system--1.5MB max.--which led me to conclude that the OS must be set up to buffer output files in ram until closed.

So if it is the OS that is responsible for the memory consumption, then it will equally affect output files being written by a seperate Perl process, or a system utility unless the OS discriminates between system utilities and other programs? Or maybe this is a build option?

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

Comment on Re^4: Managing System Memory Resources

Replies are listed 'Best First'.
Re^5: Managing System Memory Resources by dws (Chancellor) on Jun 26, 2004 at 07:51 UTC
Given the OP's observations about memory patterns, I rather suspect that the sample programs aren't representative of what's really happening. It might be as simple as no "my" in `my $str .= stuff;` [download] or it might be something deeper. If the full code isn't available for us to diagnose, and memory use is acceptable if batch sizes are reduced, I say reduce the batch sizes and concatenate.	[reply] [d/l]