timgerr1 has asked for the wisdom of the Perl Monks concerning the following question:

Hey all, I need some help, I have asked this question in other places but no one has answered it. Here I go. I have written a script that will directory walk through my Win32 file server and make sure a certain group is contained on every directory. I am traversing 5 Terabytes of data so there are many folders. The problem that I am having is that the longer I run my script the more ram it takes up. I an doing a directory walk, I am not sure if that is the problem. Thanks for the help.
# This scrip will travers a given depth of folders and see if an admin + (or user) # is not in that group. When you call the script, you have to give 3 +arguments # from the command line # 1. firs you must give a number of levels that you want to dig do +wn into # 2. You must give a starting point to that you want to search\tra +verse # 3. you give a path and file name of an output file that will be +written # to when a fooder does not have the group and correct rights # This will walk a directory when given 2 arguments, the depth and the + directory # For some reason when I run this script I get 2 lines or errors all t +he time so I # call a clear screen to clear the errors. It has somting to do when +I call the # Win32::Perms and File::DirWalk use File::DirWalk; use Win32::Perms; # Clear the screen system("cls"); # Setting up the sub rutiene sub getperms; sub ERROR; # Getting command line arguments my($arg1,$arg2,$arg3) = @ARGV; chomp($arg1,$arg2,$arg3); # changing the color to regular in the command prompt system("color 07"); # Making sure that all the arguments are filled if(length($arg1) < 1 or length($arg2) <1 or length($arg3) <1) { system("color A"); print "You need to give 3 arguments\n\nThe 1st is the folder depth + that you want to search.\n\n"; print "The 2nd is the path you want to traverse\n\n"; print "The 3rd is the path and file name for the log file such as +c:\\somting_err.txt\n\n"; print "EXAMPLE somthing.pl 3 t:\\A_Directory \"c:\\program files\\ +temp\\error log.txt\"\n\n"; print "REMEMBER to add quotes \" \" to any path if there are any s +paces in that path\n\n"; exit; } # Changing the output color system("color 4e"); print "Folder_Look_For_Admin_Recrus.exe v1.0\n"; my $x = 0; # Setting up and executing the directory walk my $dw = new File::DirWalk; my @all; $dw->onDirEnter(sub { my ($dir) = @_; getperms $dir; # if($x == 1000){ # print $dir . "\n"; # $x = 0; # } # $x++; return File::DirWalk::SUCCESS; }); # Running directory walk $dw->setDepth($arg1); $dw->walk("$arg2"); sub ERROR { my($error) = @_; chomp($error); open(WRITER, ">>". $arg3); print WRITER "lala " . $error . "\n"; close(WRITER); } # This is the perms sub sub getperms { # We need to have a collection varieble to add all the output into +. We will use this variable to parse information $strMaster = ""; my ($obj) = @_; $Perm = new Win32::Perms($obj); # print "Permissions for: $obj ,\n"; $strMaster = $obj; $read = 'READ_CONTROL SYNCHRONIZE FILE_READ_EA FILE_READ_ATTRI +BUTES'; $write = 'SYNCHRONIZE FILE_WRITE_EA FILE_WRITE_ATTRIBUTES'; $RW = 'READ_CONTROL SYNCHRONIZE FILE_READ_EA FILE_WRITE_EA +FILE_READ_ATTRIBUTES FILE_WRITE_ATTRIBUTES'; $RE = 'READ_CONTROL SYNCHRONIZE FILE_READ_EA FILE_EXECUTE F +ILE_READ_ATTRIBUTES'; $mod = 'DELETE READ_CONTROL SYNCHRONIZE FILE_READ_EA FILE_WRITE +_EA FILE_EXECUTE FILE_READ_ATTRIBUTES FILE_WRITE_ATTRIBUTES'; $full = 'STANDARD_RIGHTS_ALL FILE_READ_EA FILE_WRITE_EA FILE_E +XECUTE FILE_DELETE_CHILD FILE_READ_ATTRIBUTES FILE_WRITE_ATTRIBUTES'; $Perm->Dump(\@List); foreach $acct (@List) { my(@Permissions); next if ($acct->{Entry} ne "DACL"); Win32::Perms::DecodeMask($acct->{Mask},\@Permissions); my $final = join(" ",@Permissions); if($final =~ m/$full/) { # print ",$acct->{Account} , Full"; $strMaster = $strMaster . ",$acct->{Account} <> Full"; } elsif ($final =~ m/$read/) { # print ",$acct->{Account} , Read"; $strMaster = $strMaster . ",$acct->{Account} <> Read"; } elsif ($final =~ m/$write/) { # print ",$acct->{Account} , Write"; $strMaster = $strMaster . ",$acct->{Account} <> Write"; } elsif ($final =~ m/$RW/) { # print ",$acct->{Account} , Read Write"; $strMaster = $strMaster . ",$acct->{Account} <> Read Write"; } elsif ($final =~ m/$RE/) { # print ",$acct->{Account} , Read Execute"; $strMaster = $strMaster . ",$acct->{Account} <> Read Execute" } elsif ($final =~ m/$mod/) { # print ",$acct->{Account} , Modify"; $strMaster = $strMaster . ",$acct->{Account} <> Modify"; } else { # print ",$acct->{Account} , SPECIAL <--> $final"; $strMaster = $strMaster . ",$acct->{Account} <> SPECIAL <--> $ +final"; } } # Change this search to serch for the group that you are looking f +or if($strMaster !~ m/Administrators <> Full/){ # <---------------- +----- Change this group, this is the group that you want to search fo +r # Print error line to log file ERROR $strMaster; print $strMaster . "\n"; } } print ("\n\nThe End\n\n"); system("color e");

Replies are listed 'Best First'.
Re: Memory Leaks can it be helped?
by BrowserUk (Patriarch) on Aug 17, 2007 at 19:41 UTC
Re: Memory Leaks can it be helped?
by cdarke (Prior) on Aug 17, 2007 at 19:33 UTC
    If you read the rating comments on CPAN you will see that several users have issues with File::DirWalk. For all I know it might be a perfectly good module, but you might be better with the core module File::Find, which has probably had a lot more testing.
    And you certainly should not ignore error messages just because you do not understand them - error messages are your friends! Ask about them here if necessary.
    And:
    use strict; use warnings;
Re: Memory Leaks can it be helped?
by dogz007 (Scribe) on Aug 17, 2007 at 19:34 UTC
    The increasing amount of memory usage does not necessarily suggest that you have a leak (although you should strive to fix the errors you mentioned). It will be quite natural to watch the memory usage increase, since the DirWalk object you've used is constantly storing up some portion of your directory tree. The longer it searches, the larger this stored tree gets. However, you should be careful about choosing a search method for walking directories. Your fileserver hopefully has some final depth, so a depth-first-search (DFS) should not run off on you like it would if you were traversing the internet. A breadth-first-search (BFS) will most likely take more memory since your file structure has a finite depth. Check to see which type of search method your object is using and whether or not can specify your preference. If it cannot, and you don't mind experimenting a little, you could use some of the dirWalk iterator techniques discussed in Higher Order Perl. (I think after I get home from work, I'll post some examples of both DFS and BFS home-grown dirWalkers.)
Re: Memory Leaks can it be helped?
by chromatic (Archbishop) on Aug 17, 2007 at 19:17 UTC
    For some reason when I run this script I get 2 lines or errors all the time so I call a clear screen to clear the errors. It has somting to do when I call the Win32::Perms and File::DirWalk

    I think there might be a relation between these errors and the memory use. I also note that you're using globals for some of the data that doesn't need to persist. If you were to use properly-scoped lexicals, you may see less memory use.

      I think there might be a relation between these errors and the memory use.
      What makes you think that? It turns out that both modules export a constant named SUCCESS, and that is the source of the warnings. But it has nothing to do with the memory leak. The OP is using the fully qualified name $File::DirWalk::SUCCESS, so there's no need to import anything from that module, and "use File::Dirwalk ();" should squelch the warnings.

      That said, I agree with some of the reviewers of File::DirWalk in that I don't see any advantage of this module over File::Find.

        What makes you think that?

        Years of experience answering questions where people don't post the warnings that, if they had read and understood, would have helped them fix their own problems, combined with the curious mixture of global and lexicals and the possible circular reference involved with passing one object to another.

        Call it a hunch.