SkullOne has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a perl script that does some basic file operations, in this case moving files around based on their creation date.
I have a large folder with 60,000+ files inside, and when I run this particular Perl script, it exits without any output and no errors.
Is there a memory limit or something I could be running into?
Heres a snippet:
for  my $file (<*.txt>)
  { 
      $count = $count + 1;
      
      my ($name,$path,$suffix) = fileparse($file,"\.txt") ;
    my $info = stat($file);
    my $datestamp = strftime(".%Y%m%d", localtime($info->mtime));
    my $datestamp2 = strftime("%Y%m%d", localtime($info->mtime));
    #print "\nMaking Directory if it doesnt exist: $dst2\\$datestamp2"
+;
    mkdir "$dst2\\$datestamp2" or "Error making Directory\n";
    print "\n Moving \"$file\" to >> $dst2\\$datestamp2\n";
    move $file,"$dst2\\$datestamp2\\$name$suffix" or warn "Cannot copy
+ $file $!\n" ;
    if($count > $limit)
    {exit;}
  }

[download]

Replies are listed 'Best First'.
Re: Perl file operations
by kyle (Abbot) on Feb 23, 2008 at 03:22 UTC

    I did perl -le 'print for <*>' | wc -l, and it listed out over 100000 files (after a while). I don't think the size of your directory is the problem. It still could be the problem if your memory is very limited, but I'd expect you to get a message about that explicitly if that were the case. (The file globbing you do builds up a list of all those 60000+ files in memory at once.)

    The problem may be that after mkdir, you or into a string. The string doesn't do anything. You may want: 

    mkdir "$dst2\\$datestamp2"
    or die "Can't mkdir '$dst2\\$datestamp2': $!\n";

    [download]

    After that, I expect the move to fail too, but the warn there should be telling you why. If it's not, I wonder if STDERR is being redirected, or there's a $SIG{__WARN__} handler or something.

    I'd make your end-on-limit more explosive: 

    if ( $count > $limit ) {
    die "limit exceeded: $count files processed";
}

    [download]

    Generally I recommend setting $limit to about 10 and sprinkle a lot of print statements in the loop to make sure all those variables have the values you think they do.

[reply]
[d/l]
[select]
Re: Perl file operations
by jwkrahn (Abbot) on Feb 23, 2008 at 03:23 UTC

    While I don't have a directory with 60,000 files in it to test your code I would guess that  <*.txt> is creating such a large list that it runs out of memory.    Try using opendir and then readdir in a while loop to cut down on memory usage: 

    opendir my $DH, '.' or die "Cannot open the current directory: $!";

while ( my $file = readdir $DH ) {
    next if '.txt' ne substr $file, -4;

    [download]
[reply]
[d/l]
[select]
      While I don't have a directory with 60,000 files in it to test your code I would guess that <*.txt> is creating such a large list that it runs out of memory.

      At an estimate of 64 bytes of overhead per SV, 60,000 strings produced by that list will take up 3.6 megabytes of memory, not counting the lengths of the filenames.

[reply]
[d/l]
        Monks,
        I am stupid. The file extensions in the directory were different that I was expecting.
        I fixed the file extension, and it slowly but surely processed the 60,000+ files into their respective directories. I watched memory usage on the successfull run, and it hovered between 60 and 70 megabytes, using Activeperl.
        Please dont harm me :(
[reply]

Back to Seekers of Perl Wisdom