Firstly, I don't really see the need to run through your log file more than once. Also, I think just a single hash (a HOH, actually) would suffice for your purposes. I would use the originating message ID as the key to the hash, and then have several sub-keys such as "destination_id", "status", "source_host", "destination_host", etc. And then just populate the values for each of these as you run through the log file. Once you are done, it's just a matter of iterating through your hash keys and outputting those of interest.

The second point is yes, it would be better to use a while loop to read through your log file. By using a foreach loop, you are effectively slurping the whole file into memory. Whereas a while loop will just read line by line. So something like:

while (my $line = <FH>) {
    chomp($line);   # you probably want to do this
    # do whatever with contents of $line
}
[download]

An HoH is one of those magical incantations I have not got the hang of yet. But after your suggesting I could just go through the log once, and use while loops in preference (I didn't know that a foreach grabs everything into memory), I managed to nut out a way of grabbing the right info at the right time (I've stuck my mucky code at the bottom of the thread). Thanks very much for your input; it was a great help.

foreach $keys (%msgids) {
            if ($key =~ /$line/) {
                delete $msgids $key;
            }
      }
[download]

foreach $key (keys %msgids) {
            if ($key =~ /$line/) {
                delete $msgids{$key}
            }
      }
[download]

while (($key, $value) = each(%msgids)) {
            if ($key =~ /$line/) {
                delete $msgids{$key}
            }
      }
[download]

Ack, yes, my grasp of hashes is shaky at best, and it helps if I use the correct syntax! That while loop syntax is great - it looks much better, and I made heavy use of it (in my ugly script below). Thanks for the help!

Thanks to the excellent input I received, I made something that appears to work - it may be ugly, but it's saved me a big job. The interesting part is that the client host (originating server) is only mentioned with reference to the first message ID, and the external delivery status only goes with the second message ID. The line that has both IDs comes halfway through the logging sequence (and other unrelated message transactions are interpolated in between).

#!/usr/bin/perl
use warnings;
use strict;
my @logfiles = qw(maillog.4 maillog.5 maillog.6 [..]);
my $line;
my $file;
my $key;
my $value;

open (CLIENTOUT, ">>client.txt") or die "Cannot open output file: $!";

for $file ( @logfiles )  {
    open (FH, $file ) or  die "Cannot open file: $!";
    print "$file\n";
    my %msgids;
    while (my $line = <FH>) {
        chomp($line);
        if  ( $line =~ /client=(ourhost1|ourhost2)/ ) {
            no warnings 'uninitialized';   
#I don't care about warnings about something uninitialised in line 23
            my @id = ($line =~ /\_([0-9A-F]{11,12})/);
            $msgids{ $id[0] } = ();
        }
        if ( $line =~ /to=<.*\.gov>.*relay=localhost/ ) {
            if ( $line !~ /to=<.*(ourdomain\.gov)/i) {
                while ( ($key, $value) = each(%msgids) ) {
                    if ($line =~ /$key/) {
                        my @id = ($line =~ /\ ([0-9A-F]{11,12})/g);
                        $msgids{$key} = $id[1];
                    }
                }
            }
        }
        if ( $line =~ /to=<.*\.gov*.status=sent/ ) {
            if ( $line !~ /to=<.*(ourdomain\.gov|relay=localhost)/i) {
                while ( ($key, $value) = each(%msgids) ) {
                    if  ( defined $value ) {
                        if ($line =~ /$value/) {
                            print CLIENTOUT "$line\n";
                        }
                    }
                }
            }
        }
    }
    close FH;
}
[download]

(Moving this off my pad and into here - where it belongs)

Try running the following code. I think it meets most of your requirements. To be perfectly honest, it's probably not much better than your existing code - but it does demonstrate a different approach, and the use of a HOH which I mentioned in my initial reply.

If there are any parts of it that need explaining, feel free to ask :)

#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;

# Replace the following with the location of your maillog
# (Don't forget to include the full path if it isn't in the cwd)
my $mail_log = 'billie_t.log';

my %messages;
my $valid_src = 'exch';
my $valid_dst = 'gov.au';
my $verbose = 1;    # Change this to 1 if you want to see which lines 
+are ignored

open my $mail_fh, '<', $mail_log
    or die "Cannot open $mail_log:$!\n";


while (my $line = <$mail_fh>) {
    chomp($line);
    if ($line =~ /qmgr.*?from=/) {      # It's a new mail
        my ($id, $from) = (split /\s+/, $line)[5,6];
        chop($id);                      # remove the colon
        $from =~ s/^from=<(.*?)>,/$1/;
        $messages{$id}{from} = $from;
    }
    elsif ($line =~ /client=/) {
        my ($id, $client) = (split /\s+/, $line)[5,6];
        chop($id);
        $client =~ s/client=(.*)/$1/;
        $messages{$id}{client} = $client;
    }
    elsif ($line =~ /relay=localhost/) {
        my ($id, $to, $status) = (split /\s+/, $line, 10)[5,6,9];
        chop($id);
        $to =~ s/to=<(.*?)>,/$1/;
        my ($remote_id) = $status =~ m/([0-9A-Z]+)\)$/;
        $status =~ s/.*?\((.*?)\)$/$1/;
        $messages{$id}{to} = $to;
        $messages{$id}{status} = $status;
        $messages{$id}{remote_id} = $remote_id;
    }
    else {
        print "Skipping... $line\n" if $verbose;
    }
}

for my $id (sort keys %messages) {
    no warnings 'uninitialized';
    next if (!$messages{$id}{client} || $messages{$id}{client} !~ /\Q$
+valid_src\E/);
    next if (!$messages{$id}{to} || $messages{$id}{to} !~ /\Q$valid_ds
+t\E/);
    print qq(Valid message found: ID=$id\n\tFROM=$messages{$id}{from}\
+n\t),
          qq(TO=$messages{$id}{to}\n\tSTATUS=$messages{$id}{status}\n\
+t),
          qq(CLIENT=$messages{$id}{client}\n\tREMOTE_ID=$messages{$id}
+{remote_id}\n\n);
}

# Uncomment the following line to get a better picture of the datastru
+cture that is created
# (You might want to redirect the output to a file)
#print Dumper(\%messages);

close $mail_fh;
[download]