Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Virtual memory issue with File::Find?

by OzzyOsbourne (Chaplain)
on Jul 30, 2001 at 15:34 UTC ( [id://100834]=perlquestion: print w/replies, xml ) Need Help??

OzzyOsbourne has asked for the wisdom of the Perl Monks concerning the following question:

I ran the following code on a w2k system on Friday. It looks like it died on Saturday night about 1/3 way through. Why, I don't know, although the drive I'm running it against is massive. The system had a low virtual memory error when I came in this morning.

I'm writing out to a text file immediately, so I don't know how there was a virtual memory problem. Can anyone else see the issue?

Thanks.

# Finds the owners of files on the share, finds their size, and prints + them to globalowners.txt # 7/27/01 # Prints [owner][file][file size (KB)][acess time code][last file acce +ss][modify time code][last modified date] # To effectively sort times, sort by the timecodes. # Import the text file to excel for best results use strict; use Win32::Perms; use File::Find; my $dir1='//server/share'; open OUT, ">shareowners.txt"; print OUT "Owner\tFile\tSize(KB)\tAccessTimeCode\tLast Accessed\tModif +yTimeCode\tLast Modified\n"; find ({wanted => \&wanted, no_chdir=>1}, $dir1); close OUT; sub wanted { if (-f){ my $File = new Win32::Perms("$_")||die; my $Own=$File->Owner(); my @stat=stat; my $kbytes = $stat[7]/1024; my $access = localtime($stat[8]); my $modify = localtime($stat[9]); print OUT "$Own\t$\$_\t$kbytes\t$stat[8]\t$access\t$stat[9]\t$ +modify\n"; } }

-OzzyOsbourne

Replies are listed 'Best First'.
Re: Virtual memory issue with File::Find?
by grinder (Bishop) on Jul 30, 2001 at 16:53 UTC
    A shot in the dark... maybe those Win32::Perms objects are not being reaped and/or they are causing memory leaks. Devel::Leak may help you spot what's going on. Otherwise you might want to see if you can get away with creating one Win32::Perms object and then keep feeding it new filenames.

    I also suspect you really want to say scalar localtime. Right now, all you are doing is collecting the seconds component of the epoch time of the file.

    Also, test your open and die if it doesn't succeed.

    Minor quibble: emit the filename as the last field. If you have to eyeball the file (even though I'm assuming it's being processed downstream by another prog) it will be easier on the eyes.

    --
    g r i n d e r
(tye)Re: Virtual memory issue with File::Find?
by tye (Sage) on Jul 30, 2001 at 21:13 UTC

    Well, taking out the no_chdir option would probably make it run faster and use less memory. But I'd be more suspicious of Win32::Perms having a leak than File::Find, mostly just because Win32::Perms has tons of "Perl written in C code" and that is notorious for being complicated enough that the author doesn't get the reference counts correct, even if the author of this module is rather highly regarded.

    So you have two experiments you can run immediately. Run the script as-is again and watch its memory consumption (running any Perl script on a system with little virtual memory if probably not a good idea, BTW) long enough to figure out about how fast it grows. Then kill it and rerun it with no_chdir turned off and then with the use of Win32::Perms commented out. Compare the rate at which memory consumption grows.

    BTW, you want to look at the growth of VM usage and most tools will prefer to show you "working set size". For example, if using Windows Task Manager (taskmgr), then select View then Select Columns... then check Virtual Memory Size.

            - tye (but my friends call me "Tye")
Re: Virtual memory issue with File::Find?
by oakbox (Chaplain) on Jul 30, 2001 at 16:23 UTC
    Take this with a grain of salt (I'm merely a scribe):
    Have you tried setting your script to autoflush?

    Put this somewhere toward the top of your script and run it again:

    $|=1;
    Oakbox
    "If what I'm saying doesn't make sense, that's because sense cannot be made, it's something that must be sensed"-J.S. Hall
      That's not going to help. $| only works on the currently selected filehandle which in his case is the default of STDOUT and he's not printing anything to STDOUT so $| = 1 won't do any good by itself. select(OUT); $| = 1; select(STDOUT); would autoflush OUT.
Update: Virtual memory issue with File::Find?
by OzzyOsbourne (Chaplain) on Jul 30, 2001 at 21:14 UTC

    Update2: There is a memory leak with Win32::Perms, not File::Find.

    From Dave Roth:

    Ozzy,
    It does look like the code has a memory leak. We will look into this a bit
    deeper and see if we can provide a fix.
    
    dave
    
    Dave Roth
    http://www.roth.net/
    

    -OzzyOsbourne

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://100834]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2024-03-29 13:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found