2mths has asked for the wisdom of the Perl Monks concerning the following question:

I've got a script that runs through a loop of about 500 machine names, copies a file to each, executes it and puts the returned info into a database. Trouble is the script eats memory, maybe half a Gb!

I have no idea about good practise with regard to writing memory efficient scripts. Where do I start? I have checked the Camel book and SuperSearch but not found anything that hits me in the face. That said PP does have an efficency section which mentions avoiding 'tight' loops and excessive system calls. I have a lot of both, is there anything I can do to tidy up after them?

TIA

Oh and I ought to have said I have no idea whether my loops are 'tight' or not but there are lots of loops

Ok, I think you deserve a laugh, here's a cut down (slightly cut down) version of the code.
#! d:/perl/bin/perl.exe -w use Win32::ODBC; use File::ReadBackwards; use Net::Ping; if ($ARGV[0]) { Use this info somewhere in the script as variable $lik +e } else { $like = '1' } open (LOG, ">>PathToLog.txt"); &date(); ### # Get latest AV version ## if ($db = new Win32::ODBC("SWDHAP301")) {} else {die "Could not connect to database via DSN SWDHAP301\n"} $db->Sql("Query to find packageID of AntiVirus software package"); while ($db->FetchRow()) { Get result of query; Put sought after info into $ver; } $db->Close(); ### # Get name of next machines to try and install package on ## $db = new Win32::ODBC("avXtra"); $db->Sql("Query to get all machines requiring install"); while ($db->FetchRow()) { $var = $db->Data ; @servers = (@servers, $var); } $db->Close(); $no = @servers; print LOG "# STARTING SCRIPT @ $datetime\tetc etc #"; ### For all machines in list as requring install, attempt to install t +hem. LOOP: foreach $server (@servers){ $verRead = $avv = ''; print "\n$time\t$server"; &date(); my $p = Net::Ping->new(); $p->hires(); ($ret, $duration, $ip) = $p->ping($server, 5.5); if ($ret) { printf("\n[ip: $ip] alive (packet return time: %.2f ms)", 1000 * $ +duration); print " \n\tChecking current AV version in anplc.coe file\n"; ### CHECK IF CLIENT IS ACTUALLY UP TO DATE $path = "\\\\$server\\c\$\\anplc.coe"; if (-e "$path") { open (FILE, "$path"); @contents = <FILE>; close (FILE); } else { print "\tCouldnt open file on $server, What now?\n" } LOOP2: foreach $line (@contents) { if ($line =~ /Virus Update - .*(\d{4}).*/) { $verRead = $1 ; print "\tVersion of AV read from anplc.coe as part of chec +k was $1\n" ; last LOOP2 ; } } if ($verRead == $ver) { print "\tAV already upto date\n"; $db9 = new Win32::ODBC("avXtra"); $db9->Sql("Update db to show client now up to date"); $db9->Close(); next LOOP; } else { print "\tVersion of client read as $verRead\n\t Continuing +with script copy and install...\n"} print LOG "Something, $aVariable, something else"; $var2 = `copy D:\\Perl\\bin\\avXtra\\$PackageSourceFiles\\$fil +ename \\\\$server\\c\$\\$filename`; if ($var2=~/1 file\(s\) copied./) { print " Copied "; print LO +G "something";} else { print " Copy Failed $var2 "; print LOG "Something" ; &FAILED(Copy_Failed_Path_Not_Found); next LOOP; } print LOG "Another something including say $datetime"; $var = `D:\\GetSupport\\utils\\psexec \\\\$server -i \"c:\\$fi +lename\" -f`; print LOG "$datetime again"; if ($var=~/error code 0/) { print " Executed Succesfully "; pr +int LOG "$datetime...";} else { print " Execution Failed "; print LOG "and again ($date +time)" } ### Delete the package. Otherwise over time the users hard drive +will get full $var3 = `del \\\\$server\\c\$\\$filename`; print LOG "not funny now"; ### Update the db with the correct version as now installed $path = "\\\\$server\\c\$\\anplc.coe"; if ($bw = File::ReadBackwards->new( "$path" )){ while(($log_line = $bw->readline)){ @array = ($log_line, @array); } } LOOP: foreach $line (@array){ if ($line =~ /Virus Update - .*(\d{4}).*/) { $avv = $1 + ; last LOOP } } $db = new Win32::ODBC("avXtra"); print "\nAV_Version = $avv , lastUpdated = $SQLdateTime , Comment += ScriptInstalled WHERE machineName like $server\n"; $db->Sql("UPDATE tbl_avXtra_main SET AV_Version = '$avv' , lastUpd +ated = '$SQLdateTime' , Comment = 'ScriptInstalled' WHERE machineName + like '$server'"); $db->Close(); } # Endof PING loop else { print " Not On Network "; print LOG "\n$datetime\t$server not on network" ; $db6 = new Win32::ODBC("avXtra"); $db6->Sql("UPDATE tbl_avXtra_main SET lastUpdated = '$SQLdateT +ime' WHERE machineName like '$server'"); $db6->Close(); } } # Endof server loop close (LOG); sub date { create some variables relating to date and time here } sub FAILED { $db = new Win32::ODBC("avXtra"); $db->Sql("UPDATE tbl_avXtra_main SET lastUpdated = '$SQLdateTime' , Co +mment = '$_[0]' WHERE machineName like '$server'"); $db->Close(); } exit 0;

Replies are listed 'Best First'.
Re: Memory Leak Advice
by davido (Cardinal) on Nov 06, 2003 at 18:20 UTC
    If you could boil your code down to a dozen (give or take a few) lines that replicate the memory leak issue, and you posted that code here, we might have a prayer of helping you find the problem. Resolving the efficiency issues of tight loops is not relevant to memory leak problems. If you've got a memory leak, the loop is only the mechanism of magnifying it, not the cause of the leak itself. Remember that memory efficiency is completely different than memory leaks. Which one is it?

    Why not post your code so we don't have to guess? Often when trimming it down to an "example" version that you can post here, you'll discover the issue on your own, but if not, at least we'll have something to start with.


    Dave


    "If I had my life to live over again, I'd be a plumber." -- Albert Einstein
Re: Memory Leak Advice
by pg (Canon) on Nov 06, 2003 at 17:27 UTC

    If that’s all what you are doing, which is not quite complex, I don’t see much chance for memory leak, with a reasonable coding style.

    How do you copy the files? My guess is that you started some child process each loop(?), and didn’t waitpid() them, thus created the possibility of using up resources. This is a common error. Pure guess, as didn't see your code

Re: Memory Leak Advice
by tilly (Archbishop) on Nov 07, 2003 at 00:42 UTC
    If we're playing the guessing game, my guess is that you have used a lot of global data structures, and they always grow, you never clear them out. Just modularizing your code, using lexically scoped variables and strict.pm might help clean some of that up.

    But that's just a guess.

Re: Memory Leak Advice
by Abigail-II (Bishop) on Nov 06, 2003 at 17:33 UTC
    I think the problem is on line 17 of your code.

    Abigail

      I beg to differ. I am certain that it relates to the recursive rentry into the invisible() function initailly called from line 42.

      cheers

      tachyon

      LMAO!

      That's it, I think it's time to start an Abigail-II fan club :)

Re: Memory Leak Advice
by 2mths (Beadle) on Nov 07, 2003 at 10:03 UTC
    I think this node is probably dead now but...

    Thanks to all that replied. I appreciate a laugh even if it's at my own expense :-)

    In an effort to improve my code I have:
    * Removed all printing to a log file (I never look at it)
    * Removed redundant lines of code (where I could identify them

    I also intend to:
    * Set variables to 0 or is that '' or NULL after they've been used - ie where I read in the log file on the client machine
    * Look at my use of loops to see if it could be simplified
    * Think about using STRICT (I know I should but I am lazy)
    * Post my code (in a much better formatted way) next time I ask a dumb question.

    One final question - Is what I'm describing a memory leak, or should I have said "Excessive use of memory"

    Thanks to one and all.