joeymac has asked for the wisdom of the Perl Monks concerning the following question:

Can any of the Honorable and Wise Monks that dwell within these Gates please help me track down the memory leak in my script? Basically, it is meant to be a long-running (read: infinitely looping) application that watches a remote directory and transfers files when new ones appear. Everything works perfectly for about one hour until it terminates with an "Out of memory!" message. I have used Devel::Peak to try to track it down, but to no avail, yet. Have included use strict and fixed variable scopes (at least in so far as I don't get any warnings upon initiation). I also tried putting most of the script into a subroutine (sub Prog{} below) in the hopes that Perl would free up its memory at the end of each iteration of the loop. The part that has me most baffled, though, is that I have a VERY similar Perl script that I wrote which is doing the same thing with some different data and it is running perfectly (for about three weeks nonstop now). I apologize if the script below is bulky, I really tried to remove unnecessary stuff.

#!/usr/bin/perl use warnings; use strict; use File::Copy; use File::Basename; use List::Compare; use Net::FTP; use Net::Netrc; my $ANCHOR = $ENV{ANCHOR}; our @globals; our @ProdRec; our @rmtDirList; our @matches; our @filesToMove; our @filesToRetrv; our $host; our $userAcct; our $ftpFailDir; our $fileToGet; our $remoteDir; our $finalDir; open CONFIG, "$ANCHOR/apps/smops/config/ConfigAscatLvl2.txt" || print +"Can't open config file\n"; my $config = join " ", <CONFIG>; close CONFIG; eval $config; print "Couldn't evaluate the config file: $@\n" if $@; for (;;) { &Prog; } sub Prog { my $prodNum = $globals[2]{'numOfProducts'}; my $dirNum = $globals[2]{'numOfGFTDirs'}; for (my $i=0; $i < $dirNum; $i++) { my $gftDirName = $ProdRec[$i]{'remoteDir'}; } my $loopSleep = $globals[2]{'sleepTime'}; my $resultsFound = 0; my $workingDir = $globals[2]{'workingDir'}; opendir( DIR1, $workingDir ) || print "Cannot open working directory: $workingDir\n"; my @wrkDirList = readdir (DIR1); for (my $i=0; $i < $dirNum; $i++) { my $remoteDir = $ProdRec[$i]{'remoteDir'}; our $userAcct = $ProdRec[$i]{'userAcct'}; our $host = $ProdRec[$i]{'remoteServ'}; my @currentDirList = &ftpDirList; push (@rmtDirList, @currentDirList); } my $lc = List::Compare->new( \@rmtDirList, \@wrkDirList ); my @onlyInRmtDir = $lc->get_unique; for (my $i=0; $i < $prodNum ; $i++) { my $host = $ProdRec[$i]{'remoteServ'}; my $userAcct = $ProdRec[$i]{'userAcct'}; my $finalDir = $ProdRec[$i]{'finalDir'}; my $remoteDir = $ProdRec[$i]{'remoteDir'}; my $ftpFailDir = $ProdRec[$i]{'ftpFailDir'}; if ( @onlyInRmtDir > 0 ) { my $grepString = $ProdRec[$i]{'grepString'}; $resultsFound = 1; my @tempMatches = grep { /$grepString/ } @onlyInRmtDir; push (@matches, @tempMatches); } ftpFileGet(); @matches = (); foreach my $getFile (@filesToMove) { copy("$workingDir/$getFile", "$finalDir/$getFile"); } @filesToRetrv = (); @filesToMove = (); } cleanUp(); sleep $loopSleep; } sub ftpDirList { my $select = Net::Netrc->lookup($host, $userAcct); my $pass = $select->password(); my $ftp = Net::FTP->new($host); $ftp->login($userAcct, $pass); $ftp->pasv; $ftp->binary; $ftp->cwd("$remoteDir"); my @currentDirList = $ftp->ls(); $ftp->quit(); return @currentDirList; } sub ftpFileGet { my $workingDir = $globals[2]{'workingDir'}; my $select = Net::Netrc->lookup($host, $userAcct); my $pass = $select->password(); my $ftp = Net::FTP->new($host); $ftp->login($userAcct, $pass); $ftp->pasv; $ftp->binary; chdir($workingDir); foreach my $fileToGet (@matches) { $ftp->cwd("$remoteDir"); my $remoteFileSize = $ftp->size($fileToGet); my $localFileName = "$fileToGet"."$globals[1]{'transfer'}"; my $ftpReturnVar = $ftp->get($fileToGet, $localFileName); next if ! defined $ftpReturnVar; my $localFileSize = (stat "$workingDir/$localFileName")[7]; if ($remoteFileSize == $localFileSize) { push (@filesToRetrv, $ftpReturnVar); } $ftp->quit(); return @filesToMove; } sub cleanUp { my $workingDir = $globals[2]{'workingDir'}; opendir( DIR1, $workingDir ) || print "Can't open $workingDir for Cl +eanup\n"; while (my $file = readdir DIR1) { next if -d "$workingDir/$file"; my $TODAY = time; my $hrtime = $globals[2]{'maxAge'}; my $mtime = (stat "$workingDir/$file")[9]; if ($TODAY - $hrtime > $mtime) { unlink $file; } } }

Replies are listed 'Best First'.
Re: Memory leak!
by chromatic (Archbishop) on Dec 12, 2011 at 19:12 UTC

    Is this the problem?

    push (@rmtDirList, @currentDirList);

    I see nothing which removes entries from @rmtDirList.

    You could work around this by passing variables to Prog() from within your infinite loop and treating them as lexicals within the function. You're far better off doing that.


    Improve your skills with Modern Perl: the free book.

      Thanks for the advice. I emptied out some of the arrays, as can be seen in the code, but this is one I must have missed. I will add:

      @rmtDirList = ();

      along with the others and see if it has an effect on the problem.

Re: Memory leak!
by BrowserUk (Patriarch) on Dec 12, 2011 at 19:21 UTC

    And what is this loop meant to do?

    for (my $i=0; $i < $dirNum; $i++) { my $gftDirName = $ProdRec[$i]{'remoteDir'}; }

    You assign a bunch of values from a global array of hashes to a loop-local lexical scalar, and then do nothing with any of them?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      I took out the line from the full script where the directory names are printed to the a log file:

      print LOG "Retrieving data from remote dir: $gftDirName\n";
Re: Memory leak!
by BrowserUk (Patriarch) on Dec 12, 2011 at 19:13 UTC

    The first thing I notice is that the code you posted will not even compile. You have a missing close brace:

    sub ftpFileGet { my $workingDir = $globals[2]{'workingDir'}; my $select = Net::Netrc->lookup($host, $userAcct); my $pass = $select->password(); my $ftp = Net::FTP->new($host); $ftp->login($userAcct, $pass); $ftp->pasv; $ftp->binary; chdir($workingDir); foreach my $fileToGet (@matches) { $ftp->cwd("$remoteDir"); my $remoteFileSize = $ftp->size($fileToGet); my $localFileName = "$fileToGet"."$globals[1]{'transfer'}"; my $ftpReturnVar = $ftp->get($fileToGet, $localFileName); next if ! defined $ftpReturnVar; my $localFileSize = (stat "$workingDir/$localFileName")[7]; if ($remoteFileSize == $localFileSize) { push (@filesToRetrv, $ftpReturnVar); ## <<<<<<<<<<<<<<<<<<<<<<< Guess what is missing here!! } $ftp->quit(); return @filesToMove; }


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Sorry, that closing brace must have gone to the way side while I was trying to clean up the script to post here. The full code does, in fact, compile. As I said, it runs for about an hour before ending with the "Out of memory!" message.

Re: Memory leak!
by roboticus (Chancellor) on Dec 12, 2011 at 19:14 UTC

    joeymac:

    Try calling Data::Dumper on each of your global variables inside your loop, and direct the output to a text file. Then look at the results when it crashes and see which variable is unexpectedly large. Then concentrate on the relevent bits of code. There are better ways to do it, but this one's quick and easy.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

Re: Memory leak!
by cavac (Prior) on Dec 12, 2011 at 19:26 UTC

    I didn't look to closely. The indenting in your code looks a bit random (hard to follow the flow). Try perltidy to fix that. But there are a few things that you should take a look at:

    First, you call a function like this:

    for (;;) { &Prog; }
    I think you really meant
    while(1) { Prog(); }

    The function cleanUp calls opendir but not closedir. Looks like you do the same in Prog. One of the golden rules is: Always close handles as soon as you don't need them anymore.

    Don't use '#ff0000':
    use Acme::AutoColor; my $redcolor = RED();
    All colors subject to change without notice.

      If you deparse for(;;){ you get:

      perl -MO=Deparse -e"for(;;){ print 'hello' }" while (1) { print 'hello'; }

      Whilst I agree while(1) { is clearer, for(;;){ isn't actually an error.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?

        Ah, thanks. Didn't realize Perl already does that. Good catch!

        BREW /very/strong/coffee HTTP/1.1
        Host: goodmorning.example.com
        
        418 I'm a teapot

        for(;;){ isn't actually an error.

        This is intentional. In both C and Perl, any combination of the three expressions can be omitted.

        • If the initialisation expression is missing, no initialisation is performed before entering the loop.
        • If the exit condition is missing, the loop doesn't exit.
        • If the counting expression is missing, no code is executed at the end of every pass.

      Thanks for the advice on the closedir. I will add these in and see if it has an effect on the problem. Just for reference, is there much difference between infinitely looping using a

      while(1) {}

      loop rather than a

      for (;;){}

      loop?

        The first one makes it a bit clearer that you are intentionally looping an infinite number.

        I'm not sure about this, but the while loop may also be a tiny bit more efficient in this case.

        BREW /very/strong/coffee HTTP/1.1
        Host: goodmorning.example.com
        
        418 I'm a teapot
        No, while (1) {} gets optimised to for (;;) {} (which I use and pronounced "for ever").
Re: Memory leak!
by joeymac (Acolyte) on Dec 13, 2011 at 13:52 UTC

    If I may be so bold as to reel everyone back in to the original question, my script still seems to be running out of memory after about an hour. If anyone would like to continue to (needlessly) debate the differences between for(;;) and while(1), go to my previously posted question Infinite Loop Question and go wild. I have tried:

    closedir DIR1;

    in both subroutines after I opened and read directories, and I also tried emptying out the @rmtDirList at the end of the Prog subroutine:

    @rmtDirList = ();

    but neither has worked yet. As I said, it is stumping me because a very similar version of the script has been running nonstop for about a three weeks now, and I never ran into this problem when developing it. In fact I am testing the problem script on exactly the same server machine that the other is currently running! Any other insights or suggestions would be greatly appreciated. Thanks!!

      @rmtDirList isn't the only global I see you modify in place without clearing. Try also my other suggestion of passing in global variables, copying their values into lexicals, and leaving the globals unmodified. That's far easier in the long run than pairing up pushes and pops every place you modify a global.


      Improve your skills with Modern Perl: the free book.