Perl is a memory eater. But in my experience you only get to these huge amounts of memory footprint if you have large data structures holding small items. Eg, a million array items of one byte will take 43 Mb of memory. Array but more so hash overhead is large.

To me it sounds like you are trying to rebuild your PostgreSQL database in perl memory. That would be a bad idea. If this is the case and if you did so because Postgres communication is too slow (I doubt that with 10/s) move to a faster system, like BerkeleyDB (www.sleepycat.com).

About the malloc system, perl normally comes with its own. If you want to know more about it, read perlguts.

HTH,

Jeroen
"We are not alone"(FZ)

Hi,

Thanks for your observations, but due to a NDA in my project I can't publish any code. But looking the memory consumption I can see that it grows about 1 MB/sec. I think the problem is in this part :

	my $config = Config->new(config_file => "config.xml", debug=>0);





	$bd =  db_out->new ($config->{db_data});





	while (1) { 





  		$records = $bd->stack_out_get(0);





  		if(defined($records)) {





    			while (($key,$msg)= each %$msg ){	





      			process_msg($bd,$msg);





    			}





  		} 





	}

Sorry, stack_out_get returns a hash with all the resultset of de query, 10 records each like
id|TXT VARCHAR(2048)|timestamp_a|timestamp_b|app_Code VARCHAR(3)|number (integer)

TXT lenght it's alway about 250 char, but in a future version could be greater.

First of all let me say that I dearly hope that the conditional in this while loop has a typographical error:

while(($key, $msg) = each %$msg) {
    process_msg($bs,$msg);
}
[download]

I hope you mean:

while(($key, $msg) = each %$record)
[download]

If you don't mean this, then this might be some of your problem. Try renaming your hash value.

You can also make this much easier to read by using a foreach loop (if you only want to handle the msg):

foreach my $msg (values %$records) {
    process_msg($bd, $msg);
}
[download]

Anyway, I wanted to suggest that you look at Perl's profiling package. It can't exactly help you find out where all your memory is going, but it might help you track down where it's being lost. (Note that you should definately be using "my" for each of your variables to ensure that they don't hang around out of scope.)

You can read about Perl's profiling package with perldoc Devel::DProf

The shorts of it are to profile a script run the perl interpreter with the -d switch. So perl -d:DProf test.pl. when it's done, check the results by running dprofpp or dprofpp -T.

I hope that helps. :)

Does the program grow if you leave out that process_msg subroutine call?

Also, use strict and my variables where possible, and don't make circular references like $a = \$b; $b = \$a;, or they'll never really get out of scope (they might go out of scope and be unreachable, but since they refer to eachother, they'll never go away)

Finally,If you haven't already done so, check out perldebguts debugging memory section.

mitd-Made in the Dark
'Interactive! Paper tape is interactive!
If you don't believe me I can show you my paper cut scars!'

What's probably happening is that you have some data which isn't properly scoped so it's not getting cleaned up. If you can post some of your code, people here may be able to help. Make sure your are using strict and warnings to catch accidents.

If you have really huge data structures that you have to process, you could look for a way to swap speed for memory consumption by storing parts of it on disk while you work on it. (Storable is good for this.) Obviously this doesn't work with an approach that requires one large structure.

I think you definitely have a memory leak. Please observe the pattern of memory usage. If it grows slowly, a leak is the most likely cause.

As others said, if you post your code (or the relevant parts of it), probably a lot of help can result :)

Good luck.