Limbic~Region has asked for the wisdom of the Perl Monks concerning the following question:

All:
A specific type of mail message is causing an MTA downstream from me to "blow up". There is nothing wrong with the messages themselves, but the MTA requires an upgrade to handle these messages that requires testing before they can implement. Unfortunately, due to the Holiday season, this isn't happening in a timely manner. Here is the information:

  • All of the message file names begin with di
  • If the string 30 nulls followed by a tab followed by two nulls followed by followed by left paranthesis "(" followed by 1 null appears 10 or more times, the message is "bad" The messages need to be "moved" to another directory, so they won't be processed until after the upgrade
  • There is a file called "exclude", which is a list of di files that should not be processed regardless of the above requirement
  • The queue directory with the di files is very transient, so speed is of the essence
  • The exception list changes much less frequently
  • Sleeping between passes through the directory is critical as to not waste system resources

    Any and all help would be appreciated as the folks downstream have begun 24X7 coverage.

    Here is what I started using - thanks to everyone in the CB

    #!/usr/bin/ksh while : do cd <dir> perl -0777 -ne '$foo=s/\0{30}\t\0\0\(\0//g;rename $ARGV,"../capture/ +n ested/$ARGV" if ($foo >= 10)' di* sleep 3 done

    The problem with this is that it doesn't provide for an exception list.
    Thanks in advance,
    Limbic~Region

  • Replies are listed 'Best First'.
    (tye)Re: Hitting a moving target (transient files)
    by tye (Sage) on Dec 18, 2002 at 20:03 UTC

      Try this:

      #!/usr/bin/perl -w use strict; my $Directory= "/var/spool/mail"; my $BadDir= "bad/"; my $BadString= ("\0"x30)."\t\0\0(\0"; my $BadCount= 10; my $ExceptionsFile= "exclude"; my $ExceptionsUpdated; my %Exception; my $SleepLength= 1; chdir( $Directory ) or die "Can't chdir to $Directory: $!\n"; opendir DIR, "." or die "Can't open directory, $Directory: $!\n"; $/= $BadString; while( 1 ) { if( $ExceptionsUpdated != -M $ExceptionsFile ) { open EXCEPT, "< $ExceptionsFile" or die "Can't read $ExceptionsFile: $!\n"; my @except= <EXCEPT>; close EXCEPT; chomp( @except ); @Exception{@except}= (1) x @except; } rewinddir( DIR ) or die "Can't rewind directory, $Directory: $!\n"; my $file; while( $file= readdir(DIR) ) { if( $file =~ /^di/ && ! $Exception{$file} ) { if( ! open FILE, "< $file" ) { warn "Can't read $file: $!\n"; next; } my $line; for( 1..$BadCount ) { last unless defined( $line= <FILE> ); } close FILE; next if ! $line || ! chomp($line); if( ! rename( $file, $BadDir.$file ) ) { warn "Can't rename $file to $BadDir$file: $!\n"; } else { warn "Moved $file to $BadDir$file.\n"; } } } sleep $SleepLength; }

              - tye
    Re: Hitting a moving target (transient files)
    by waswas-fng (Curate) on Dec 18, 2002 at 19:55 UTC
      If this is sendmail you should be able to do a Milter to do this without trying to race queues. or even easier is to make the sendmail listener on 25 a no deliver and use the queue mover script included in contrib to move queue files from the no deliver queue to an active deliver queue (testing them for your bad messages at that time). Any questions let me know.

      Edited:
      I don't know that I got my point across, most MTA's use queues as very transient stores for files, unless you force the queuing system to hold a file and then force deliverability after you are some scanning the file you are in a race with the MTA. This means you may pass the messages without scanning at all, clobber messages that are being created or delivered or cause an issue with the MTA if you have the file opened while it is trying to delete. Almost all mailing systems (sendmail, qmail,exim etc) have ways to create an accepting queue that listens on port 25 and queues incoming messages, but then holds them in a queue without delivering. You than can check the lock stat of the mail message act on it and then move it to a queue that _is_ actively delivered. For an example of moving sendmail messages between such queues check out re-mqueue.pl in the contrib dir of the source tarball. other MTA's are a bit different but the same general idea works. You will save yourself a lot of headaches doing this the right way instead of just scanning live transient files. My .02...

      -Waswas
    Re: Hitting a moving target (transient files)
    by talexb (Chancellor) on Dec 18, 2002 at 19:59 UTC

      About the exception list (from a comment in CB): Add something to the condition where you move (rename) the file that prevents the move if the file is on an exception list.

      Because it looks like this is a ksh script I can't offer any code, sorry.

      --t. alex
      but my friends call me T.
    Re: Hitting a moving target (transient files)
    by Limbic~Region (Chancellor) on Dec 18, 2002 at 22:44 UTC
      All:
      I want to thank everyone that gave me input. tye wrote a complete program within minutes. The problem was that it wasn't fast enough even though it has much better error handling. I came up with this:

      #!/usr/bin/perl -w chdir "<directory>"; my @excludelist; my $excludefile = "<directory>/exclude"; my $count; while (1) { if ( -s $excludefile ) { open (EXCLUDE,"$excludefile"); @excludelist = <EXCLUDE>; close (EXCLUDE); } undef $/; @ARGV = <di*>; while (<>) { next if (grep /\b$ARGV\b/ , @excludelist); $count = s/\0{30}\t\0\0\(\0//g; rename $ARGV,"../capture/nested/$ARGV" if ($count >= 10); } $/ = "\n"; @excludelist = (); sleep 3; }

      This screamed through a large list.

      I also wanted to address the numerous people who suggested that there should be a better way to do this other than trying to move files on a live queue. I agree that if there was a way to get my propietary MTA to recognize this pattern and take a different action on them before writing them out it would be the best solution - but alas there is not.

      This particular directory is where the messages sit once my MTA has completely processed them. What you say, your MTA is finished with them but doesn't send them along to the next MTA?

      No, this particular propietary mail system waits for the distant end to establish an FTP session and download these mail messages.

      Yes - mail being pulled via FTP

      No - no better way to do it

      Thanks again,
      L~R

        You still seem to be under the mistaken impression that "less code" means "faster". Several of your code reductions will make the code slower. The only change that I see potentially making your code much faster is using slurping (which means the files must be fairly small). Given that, I'd do:

        #!/usr/bin/perl -w use strict; my $Directory= "/var/spool/mail"; my $BadDir= "bad/"; my $BadString= ("\0"x30)."\t\0\0(\0"; my $BadCount= 10; my $ExceptionsFile= "exclude"; my $ExceptionsUpdated; my %Exception; my $SleepLength= 1; chdir( $Directory ) or die "Can't chdir to $Directory: $!\n"; opendir DIR, "." or die "Can't open directory, $Directory: $!\n"; undef $/; while( 1 ) { if( $ExceptionsUpdated != -M $ExceptionsFile ) { open EXCEPT, "< $ExceptionsFile" or die "Can't read $ExceptionsFile: $!\n"; my @except= <EXCEPT>; close EXCEPT; chomp( @except ); @Exception{@except}= (1) x @except; } rewinddir( DIR ) or die "Can't rewind directory, $Directory: $!\n"; my $file; while( $file= readdir(DIR) ) { if( $file =~ /^di/ && ! $Exception{$file} ) { if( ! open FILE, "< $file" ) { warn "Can't read $file: $!\n"; next; } my $data= <FILE>; close FILE; my $count= ()= $data =~ /\Q$BadString/g; next if $BadCount <= $count; if( ! rename( $file, $BadDir.$file ) ) { warn "Can't rename $file to $BadDir$file: $!\n"; } else { warn "Moved $file to $BadDir$file.\n"; } } } sleep $SleepLength; }

                - tye