Works on test but not in real

Win has asked for the wisdom of the Perl Monks concerning the following question:

The following program seems to run well with a test directory structure. However when running on a massive directory structure it is worringly slow. Will the program work for big directory structures?


#!/usr/bin/perl -w

use strict;
use warnings;


use File::Find::Rule;
  # find all the subdirectories of a given directory

my $directory = "G:/X/";
my $directory_label = "G:/X";

print "Start\n\n";
print "The following directories have files in with names of more than
+ 250 characters:\n\n";


my @subdirs = File::Find::Rule->directory->in( $directory_label );


foreach (@subdirs){

print "#";
  my $directory_current = $_;
  my $rule =  File::Find::Rule->new;
    
  $rule->file;
  $rule->name( '*.*' );
  $rule->name( qr/^.{250,1000}$/); 

  my @files = $rule->in( @subdirs );

  foreach(@files){
    print $_;
  }
}
[download]

Comment on Works on test but not in real Download Code

Replies are listed 'Best First'.
Re: Works on test but not in real by marto (Cardinal) on Aug 04, 2009 at 13:31 UTC
However when running on a massive directory structure it is worringly slow. Did you benchmark/profile your code to find out where any bottlenecks are? I fear we have discussed this before. Is G: a network share or local to the system running your code? Will the program work for big directory structures? Didn't you just say it was running on a 'massive directory structure'? Martin	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Works on test but not in real by ig (Vicar) on Aug 05, 2009 at 02:06 UTC
You are looping through all the directories in @subdirs and for each directory in @subdirs you are finding all the files in all the directories in @subdirs. This seems a bit redundant. Maybe it would be quicker to do the following: use strict; use warnings; use File::Find::Rule; # find all the subdirectories of a given directory my $directory = "G:/X/"; my $directory_label = "G:/X"; print "Start\n\n"; print "The following directories have files in with names of more than + 250 characters:\n\n"; my @subdirs = File::Find::Rule->directory->in( $directory_label ); foreach (@subdirs){ print "#"; my $directory_current = $_; my $rule = File::Find::Rule->new; $rule->file; $rule->name( '.' ); $rule->name( qr/^.{250,1000}$/); my @files = $rule->in( $_ ); foreach(@files){ print $_; } } [download]	[reply] [d/l]