Without being able to provide a comprehensive fix, let me throw out some tidbits that have helped me in the past when tackling this kind of problem.

• How big is your dataset?

  If it's small, maybe you can examine your data by hand and come up with a pretty good feel for the subtleties of the dataset - like this overlap. If it's bigger, then you might wind up writing small throwaway or incremental programs just to analyze the nature of the data.

  Some of the things you might find / some things to look for:

  • Is there more than one mail message sent per connection? If so, timestamps may be off, since the example you gave shows them tied to connect/disconnects. A long-lived connection could span hours, and have multiple messages. Maybe not a problem, but something to be aware of.
  • As pointed out in the Altivore project (an open-source version of the FBI's Carnivore mail snooper), SMTP connections aren't always as cut and dried as you've seen so far. Lost connections, RSET commands, and the like can botch up your parsing if you're not looking for them. In the worst case, you could wind up creating and trying to synchronize multiple state machines that understand SMTP - yech. (Although an interesting problem!)

• Along those lines, how big is the overlap problem?

  Write a small script to count how many "clean" log entries there are - sequential entries that cover an entire connection, or maybe just an entire message (MAIL from, RCPT to, DATA). Keep counts of clean vs. interleaved log sequences - maybe there's an acceptable loss if you discard the 10% which are interleaved? Depends on your specific requirements. Some code might look like:

  # Use a stack to keep track of concurrent connections
  my @connections;
  
  while (<MERLOG>) {
    # Assume that seeing "Connection" before seeing "QUIT"
    # implies an interleaved connection record.
    if (/^Connection/) {
      ++$count_of_connections;
      push (@connections, $_);
      ++$count_of_overlaps if (scalar(@connections) > 1);
    }
    pop (@connections) if /^QUIT/
  }
  
  print "$count_of_connections connections logged, $count_of_overlaps ov
  +erlapped\n";
  print "Leaving ", $count_of_connections - $count_of_overlapped, " clea
  +n connections logged.\n"
  [download]

• Assuming you've got a good sized dataset, cache data extracted from intesive queries. If it takes 20 minutes to assemble the list of all users that received mail, cache that result away somewhere, and you'll save 20 minutes (possibly 20 minutes times the N tries it takes to get something right) the next time you need that data - maybe to correlate it with senders extracted by another script. You'll have great luck using Data::Dumper for that, bundled with most modern versions of Perl.

Peace,
-McD

#!/usr/local/bin/perl -w

use strict;

#my $log = /path/to/your/log/file(s);
#you will need to edit and uncomment the above line

my @log_list;
my $line;


#open (LOG, "$log") || die "Can't open $log:$!\n";
#uncomment the above line, if you actually use this code :-)

while ($line = <DATA>) {

#change DATA to LOG or whatever filehandle you use, too!

chomp $line;
next unless ($line =~ /MAIL\s+FROM/);
push (@log_list, $line);
}

#you will probably want to replace "next unless" with "if elsif"
# in order to get the different matches that you require.

foreach my $string(@log_list) {
print "My string is $string\n";
#this print statement is simply to demonstrate the matching text.
# obviously, you will define what YOU want to do with those matches.
}

__DATA__

  Connection from 208.1.71.165, Mon Feb 12 08:54:01 2001
  Connection from 208.1.71.165, Mon Feb 12 08:54:01 2001
  Connection from 208.1.71.165, Mon Feb 12 08:54:01 2001
  EHLO ancmail02.ancestry.com
  MAIL FROM:<Reminder@MyFamilyInc.com>
  MAIL FROM:<Reminder@MyFamilyInc.com>
  MAIL FROM:<Reminder@MyFamilyInc.com>
  RCPT TO:<user3@myhost.mydomain.net>
  RCPT TO:<user3@myhost.mydomain.net>
  DATA - 249 lines, 14229 bytes.
  DATA - 217 lines, 12388 bytes.
  DATA - 217 lines, 12403 bytes.
  
__END__
[download]

You have a project, which is more involved than it seems. I suggest that you sit down, and try to pseudo-code it first.

"Tuna- I am interested in your suggestion on how to handle this situation but isn't what you propose just a shorter way of doing what I already have? "

No, because your regex doesn't work.