in reply to Hitting a moving target (transient files)

All:
I want to thank everyone that gave me input. tye wrote a complete program within minutes. The problem was that it wasn't fast enough even though it has much better error handling. I came up with this: 

#!/usr/bin/perl -w
chdir "<directory>"; 
my @excludelist;
my $excludefile = "<directory>/exclude"; 
my $count;
while (1) {
 if ( -s $excludefile ) {
  open (EXCLUDE,"$excludefile");
  @excludelist = <EXCLUDE>;
  close (EXCLUDE);
 }
 undef $/;
 @ARGV = <di*>;
 while (<>) {
  next if (grep /\b$ARGV\b/ , @excludelist);
  $count = s/\0{30}\t\0\0\(\0//g;
  rename $ARGV,"../capture/nested/$ARGV" if ($count >= 10); 
 }
 $/ = "\n";
 @excludelist = ();
 sleep 3;
}

[download]

This screamed through a large list.

I also wanted to address the numerous people who suggested that there should be a better way to do this other than trying to move files on a live queue. I agree that if there was a way to get my propietary MTA to recognize this pattern and take a different action on them before writing them out it would be the best solution - but alas there is not.

This particular directory is where the messages sit once my MTA has completely processed them. What you say, your MTA is finished with them but doesn't send them along to the next MTA?

No, this particular propietary mail system waits for the distant end to establish an FTP session and download these mail messages.

Yes - mail being pulled via FTP

No - no better way to do it

Thanks again,
L~R

Replies are listed 'Best First'.
(tye)Re2: Hitting a moving target (transient files)
by tye (Sage) on Dec 18, 2002 at 23:17 UTC

    You still seem to be under the mistaken impression that "less code" means "faster". Several of your code reductions will make the code slower. The only change that I see potentially making your code much faster is using slurping (which means the files must be fairly small). Given that, I'd do: 

    #!/usr/bin/perl -w
use strict;

my $Directory= "/var/spool/mail";
my $BadDir= "bad/";
my $BadString= ("\0"x30)."\t\0\0(\0";
my $BadCount= 10;
my $ExceptionsFile= "exclude";
my $ExceptionsUpdated;
my %Exception;
my $SleepLength= 1;

chdir( $Directory )
    or  die "Can't chdir to $Directory: $!\n";
opendir DIR, "."
    or  die "Can't open directory, $Directory: $!\n";
undef $/;
while(  1  ) {
    if(  $ExceptionsUpdated != -M $ExceptionsFile  ) {
        open EXCEPT, "< $ExceptionsFile"
            or  die "Can't read $ExceptionsFile: $!\n";
        my @except= <EXCEPT>;
        close EXCEPT;
        chomp( @except );
        @Exception{@except}= (1) x @except;
    }
    rewinddir( DIR )
        or  die "Can't rewind directory, $Directory: $!\n";
    my $file;
    while(  $file= readdir(DIR)  ) {
        if(  $file =~ /^di/  &&  ! $Exception{$file}  ) {
            if(  ! open FILE, "< $file"  ) {
                warn "Can't read $file: $!\n";
                next;
            }
            my $data= <FILE>;
            close FILE;
            my $count= ()= $data =~ /\Q$BadString/g;
            next   if  $BadCount <= $count;
            if(  ! rename( $file, $BadDir.$file )  ) {
                warn "Can't rename $file to $BadDir$file: $!\n";
            } else {
                warn "Moved $file to $BadDir$file.\n";
            }
        }
    }
    sleep $SleepLength;
}

    [download]

            - tye
[reply]
[d/l]

In Section Seekers of Perl Wisdom