kilimanjaro has asked for the wisdom of the Perl Monks concerning the following question:

Hi Here is my code but seems to consume all 4GB of my system memory. Please help 
top - 14:19:45 up 3 days, 12:33,  1 user,  load average: 4.20, 4.27, 5
+.08
Tasks:  97 total,   1 running,  96 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.0% us, 21.2% sy,  0.0% ni, 65.9% id,  0.1% wa,  0.0% hi,  0
+.8% si
Mem:   3765572k total,  3663796k used,   101776k free,    42856k buffe
+rs
Swap:  2031608k total,      368k used,  2031240k free,  3141516k cache
+d

[download]
______ 
#!/usr/bin/perl -w
use strict;
use DBI();

my $cdr_log_file = "/var/log/asterisk/my_log";
my $mysql_host = "localhost";
my $mysql_db = "user";
my $mysql_table = "tbl_data";
my $mysql_user = "user";
my $mysql_pwd = "xxx*";

#my @ARGV;
my $insert_str;
my $sth;

my $cdate=$ARGV[0];
my $clang=$ARGV[1];
my $csrc=$ARGV[2];
my $ccode=$ARGV[3];
my $cuniqueid=$ARGV[4];


# Connect to database
# print "Connecting to database...\n\n";
my $dbh = DBI->connect("DBI:mysql:database=$mysql_db;host=$mysql_host"
+,"$mysql_user","$mysql_pwd",{'RaiseError' => 1});


$insert_str = "insert into $mysql_table (data1, data2, data3, data4, d
+ata5) values (\"$cdate\", \"$clang\", \"$csrc\", \"$ccode\",  \"$cuni
+queid\");\n";
       $sth = $dbh->prepare($insert_str);
       $sth->execute();
#       print ".";


$sth->finish();
$dbh->disconnect();
exit;

[download]

Replies are listed 'Best First'.
Re: Memory Leak: Uses 3GB+
by almut (Canon) on Jan 15, 2009 at 12:52 UTC
    my code but seems to consume all 4GB

    Do you have any specific evidence that it's your program that's using all memory?  The top output you've shown is just the general system statistics part, which is not necessarily related to your program only (and as the load averages hint at, a few other processes have been running recently).

    Anyhow, some more info might help to narrow down on the problem. For example, does your program 'hang' (or take a long time) at some point?  If so, where? (inserting a couple of debugging prints after the individual statements should suffice to tell)...

    BTW, an unrelated issue: try to avoid interpolating variable data (such as $cdate etc.) into an SQL statement — use placeholders instead. Otherwise you'll risk SQL injection, should the data ever happen to come from untrusted sources...

[reply]
[d/l]
[select]
Re: Memory Leak: Uses 3GB+
by shmem (Chancellor) on Jan 15, 2009 at 13:19 UTC 
    Mem:   3765572k total,  3663796k used,   101776k free,    42856k buffe
+rs
Swap:  2031608k total,      368k used,  2031240k free,  3141516k cache
+d

    [download]

    Well, that just tells you that the memory of your system is being used, but it doesn't tell how much is e.g. file system cache or such. That said:

    Use placeholders. Learn about why. 

    $insert_str = "insert into $mysql_table (data1, data2, data3, data4, d
+ata5) values(?,?,?,?,?)";
$sth        = $dbh->prepare($insert_str);

$sth->execute($cdate, $clang, $csrc, $ccode, $cuniqueid);

    [download]

    Placeholders also avoid backslashitis.

[reply]
[d/l]
[select]
Re: Memory Leak: Uses 3GB+
by BrowserUk (Patriarch) on Jan 15, 2009 at 12:54 UTC

    What makes you think that of the 97 processes running, that one is the one consuming all the memory?

    Because on cursory inspection, I would seriously doubt that there is any way that it could be responsible.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Memory Leak: Uses 3GB+
by targetsmart (Curate) on Jan 15, 2009 at 12:58 UTC
    I think you have mis-understood the term '1 running' in the above output of top. If so, don't assume that 1 running program is always your Perl program and don't (always)assume that only your Perl program has consumed that much of memory.
[reply]
Re: Memory Leak: Uses 3GB+
by vek (Prior) on Jan 15, 2009 at 21:19 UTC

    The code you posted accepts 5 arguments on the command line, performs an insert into a database, then exits. I don't see how this could leak memory. If you had a long running process that looped over a large amount of data and you saw the memory usage of your program increase as it ran, that would indicate a leak.

    Keep an eye on just your process when running top

    -- vek --
[reply]
[d/l]
      Thanks All for your reply. Was in transit for a day. The reason why im pointing perl is because i tried to disable it from the calling script (Asterisk PBX dialplan) and reboot the system. I then run for like 12 hours and the memory usage was as low as constant 350MB. I however couldnt keep on disabling as all are running on the production server to collect IVR statistics on different node. The script process about 100,000 request a day If any more info is required i will be glad to share for resolution Kind regards Kili
[reply]

        Can you not get a full top listing that shows the memory allocated to the individual processes?

        My guess is that some or all of those other processes are dormant but non-zombied copies of your script that are not exiting. If each copy uses a modest 30MB and you have 100 of them sitting around in memory doing nothing, that will give your 3GB total memory usage.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
Re: Memory Leak: Uses 3GB+
by targetsmart (Curate) on Jan 29, 2009 at 07:19 UTC
    If would want to know which process exceeds a particular memory limit, you can use top(nevertheless), but there are some other tools available like, ps-watcherhttp://ps-watcher.sourceforge.net/ and psmon http://search.cpan.org/~nicolaw/psmon-1.29/, this won't report statistics for you, but you can set-up hooks in the configuration file to get required statistics.
    -- In accordance with the prarabdha of each, the One whose function it is to ordain makes each to act. What will not happen will never happen, whatever effort one may put forth. And what will happen will not fail to happen, however much one may seek to prevent it. This is certain. The part of wisdom therefore is to stay quiet.
[reply]
Re: Memory Leak: Uses 3GB+
by weismat (Friar) on Jan 29, 2009 at 07:31 UTC
    Which Unix flavour/distribution are using?
    On some Linux distributions in some situations top is not useful.
    Use either free -m or vmstat to get real numbers.
[reply]

Back to Seekers of Perl Wisdom