Anyway, if any of those tools happens to work as a command-line utility (and is still available, up-to-date and within your budget), you could use it via a system call (or even a pipeline open() statement) to convert files as needed within your perl script.

If that doesn't pan out, and you're just trying to extract email address strings, perhaps others can recommend a good module. I just tried Mail::Address, and it seems to work well for the following case:

• You have the RFC822 message header pretty much intact
• You pull out the (potentially multi-line) records from the header that start with "From: ", "To: ", "Cc: " and "Bcc: "
• You pass each record (the part that follows ":"), to Mail::Address->parse()

In other words, something like this (not tested on pegasus data):

use strict;
use Mail::Address;

my $hdrstring = "";
my $checknext = 0;
open( M, "pegasus-mail.file" ) or die $!;
while (<M>) {
    if ( /^(?:To|From|Cc|Bcc): (.*)/ ) {
        $checknext++;
        $hdrstring = $1;
    }
    elsif ( $checknext and /^\s+\S.*/ ) {
        $hdrstring .= $_;
    }
    else {
        $checknext = 0;
        if ( $hdrstring ) {
            my @addr = Mail::Address->parse( $hdrstring );
            for my $a ( @addr ) {
                print $a->name, " <", $a->address, ">", $/;
            }
            $hdrstring = "";
        }
    }
}
[download]

Contact markov or Mark Overmeer for more info.

Liz

Yes, it makes sense to write such an extension: I welcome all contributions. I am not familiar with Pegasus mail, but will support anyone who tries to implement it.

MailBox uses OODoc, which means that it is simpler to write consistent documentation. If you decide to start coding based on an existing folder implementation, please ask me for a "raw" pm file (which contains both the OODoc POD and the code).

Volunteers?

First off, there are good reasons why Pegasus Mail uses its own format. Pegasus Mail's folders support some things that would not be possible with mbox format and are in some ways more robust. These days a mail directory format (e.g., nnml) would probably be better, but at the time the pmail folder format was designed, that would have been impractical for a number of reasons.

There is not a large overlap between Perl users and pmail users, but there may be a few. I have used Pegasus Mail myself in the past, before I learned Gnus, and might be interested in collaborating with you on creating a parser for pmail folders.

For just parsing, you can completely ignore the .pmi files; those are indices. (They might improve performance, but it's probably not worth figuring out a second file format just for that.) If you wanted to *write* pmail folders, then you would either have to update the indices, or delete them, the former being much preferable since deleting them would cause pmail to have to regenerate them next time it starts, which would cause a quite user-noticeable wait. But for just parsing, which sounds like a good initial goal, this is not a consideration.

The good news about the format is that most of the binary-encoded information is stuff you probably don't need, if you aren't trying to clone Pegasus Mail. Labels and flags and annotations and things. The bodies are stored *mostly* unaltered, though of course there are some provisions to prevent any specific character's presense in a message from causing problems, so messages with characters outside the printable ASCII range are encoded in some fashion (if they weren't already for transport over SMTP, though they ought to have been, in theory).

I have a number of pmail folders myself and, as I said, have some interest in working on a perl module for this. I am very unlikely to get around to doing it on my own, however.

As far as a formal specification of the format, I do not believe one has ever existed. I suppose that David Harris is using the source code or comments in the source code as his documentation, and he is unwilling to let out the source without an NDA. (This also is why there is no Linux port. David is ammenable in theory to the idea of someone doing such a port, but they would have to sign an NDA and meet other criteria. An unusual attitude for a freeware developer, perhaps, but Pegasus is of unusual quality, as well. I have switched to Gnus, because I was unwilling to be tied to a single platform and not skilled in C/C++ to attempt the port, even if I were willing to sign an NDA (something I have yet to think about enough to decide; I would have thought about it for pmail, if it were written in a language I would be comfortable working with to do a port).) So, we would have to reverse-engineer the format.

$;=sub{$/};@;=map{my($a,$b)=($_,$;);$;=sub{$a.$b->()}}
split//,".rekcah lreP rehtona tsuJ";$\=$ ;->();print$/
[download]

#!C:/Perl/bin/Perl.exe -w
use strict;
use Data::Dumper;

my %MailList;  # A list of all seen addresses.
               # This hash is setup in step 1 in the
               # previous post.

# Parse the mail box
while (<DATA>) {
    next unless /^From:\s/; # ignore all other lines
    chomp;                  # except the from line

    # assume that email addresses are separated
    # by the ',' character.
    my @emails = split /,\s*/, substr($_, 6);

    # build a hash from the email address list
    my %emails = map {
                   s/\s+$//g;  # remove tailing spaces
                   my ($name, $address);
                   if (/>$/) { # look for < ... >
                     ($name, $address) = /^(.*)\s<(.*)>$/;
                   } else {
                     $name = $address = $_;
                   }
                   $address => $name
                 } @emails;

    # inspect captured email details
    # print Dumper(\%emails);

    # check if the email address(es) are already seen
    foreach (keys %emails) {
        if (! exists $MailList{$_}) {
            # ok, we haven't seen this Email address yet
            $MailList{$_} = $emails{$_};
        }
    }
}

print Data::Dumper->Dump([\%MailList], ['MailList']);

__DATA__
From: Roger <roger@somewhere.com>, Roger 2 <roger2@nowhere.com>
To: Peter R <peterr@us.com>
Subject: Hello
Hello Peter, this from address has two addresses

From: roger3@somewhere.com
To: Peter R <peterr@us.com>
Subject: Hello
Hello Peter, this from address has no full name

From: roger3@somewhere.com
To: Peter R <peterr@us.com>
Subject: Hello
Hello Peter, has already seen this from address
[download]

$MailList = {
    'roger2@nowhere.com' => 'Roger 2',
    'roger3@somewhere.com' => 'roger3@somewhere.com',
    'roger@somewhere.com' => 'Roger'
            };
[download]

Full Name <email.address@somewhere.com>
Email.Address@somewhere.com
[download]

I have written a simple email address capture program, without using any Mail:* modules. Provided that the Emails do not have the "From: " line in the subjects, the following should work fine.

Yes, it did, thanks, and I was surprised at how out of date the distribution list is. Considering the mail boxes are more than 100Mb, perl was very quick to do the job, less than 30 seconds I'd say. No doubt it will take marginally longer to look at all the records (rather than just the "From" lines); I'd need to do this because sometimes people advise of new addresses in the body portion. I put it in a sub as you suggested. Your interim solution is ideal for my purposes, and helped me learn how to parse through files, looking for specifics (I find the Windows 'find' quite frustrating, as it won't look for multiple strings).

Thanks,

Peter

Yes, most of the 'conversion' tools for Pegasus are for address book conversion, and none of the ones I previously looked for said "Pegasus --> Unix". However, following the link you supplied and doing a bit more research, I finally realised that Eudora mailboxes are Unix format, and there _are_ a few tools to go either way (Eudora<-->Pegasus), so that means if one converted a Pegasus mailbox to Eudora, we end up with a Unix mailbox. :)

I didn't find anything useful about the Pegasus file format (but did find some indications that the format specs would not be open to public inspection).

I've found the Pegasus discussion list very active in the past, and very helpful, and would be surprised if someone wouldn't know the layout. That said, I had my own look at them, as follows:

1. The first 96 bytes contains:
   * Folder description, right filled with lowecase 'a'
   * Other folder/hiearchy info
   * More lowercase 'a' to pad to the right
2. Then the format appears (to me) to be the same as Unix
3. The message termination seems to be indicated by:
   * hex OD  OA  OD  OA  1A
   *     CR  LF  CR  LF  ->
4. From '3', it is only the hex '1A' value that could indicate the msg
+ terminator, because email bodies could have two CR/LF's anywhere.
5. I don't know if any encoding would contain hex '1A" though, so look
+ing for the 5 character string (in hex) would be safer, plus looking 
+at the next line (record) in the mailbox, it would be something like 
+"X-Apparently-To: "
[download]

Some other links I found on the conversion tools side are:

http://www.andtechnologies.com/free.html
http://www.interguru.com/MailInformation.htm
http://www.dragon-it.co.uk/pegasus.htm

You mentioned Mail::Address would be good, considering some dependencies. It's just that the Pegasus format is a _bit_ different, really just the first 96 bytes to ignore, and sort out the msg terminator. Hmm, my apologies, I just tried your sample code on a Pegasus mailbox and it dumped (printed) all the email names/address, thanks.

I see what you mean by multi-token address, addresses that are of many formats, and ones that are split over two lines; no doubt this can be a problem. :)

Peter

So, if you are fairly confident about the character string that marks the end of each mail message in this file format, then the perl code that would step through a file one message at a time would be pretty simple:

{
    open( IN, "pegasus.file" ) or die $!;

    local $/ = "\x1a";  # change the input record separator

    while (<IN>)    # $_ now contains one whole message,
    {               #   up to and including \x1a
       # do what you want with the message
       # including, if you like:
 
       @lines = split /\n/;   # work on lines

       # or you could strip off the pegasus header stuff
       # and treat the remainder as if it were vanilla email
       # (just like you'd get with common unix mail tools)
    }
}
# record separator now reverts back to default
[download]

Hi,

Thanks, that code worked okay. I think I now need to do some serious reading up on working with arrays and strings. :)

Peter

Maybe it makes sense to apply your acquired knowledge on Pegasus mailboxes to Mail::Box, so you would be able to make use of all of the nice features of Mail::Box, and the rest of us would be able to use Pegasus mailboxes with Mail::Box as well.

Yes, that's a good idea, I'm sure it would be a good extenstion for Mail::Box, possibly passing the mailbox "type" to indicate the mailbox is of Pegasus format.

Peter

You certainly have a good knowledge of the Pegasus product, and I see you use it also

There is not a large overlap between Perl users and pmail users, but there may be a few. I have used Pegasus Mail myself in the past, before I learned Gnus, and might be interested in collaborating with you on creating a parser for pmail folders.

Possibly working with Mark Overmeer , in extending the Mail::Box module to include using Pegasus mailboxes

For just parsing, you can completely ignore the .pmi files; those are indices. (They might improve performance, but it's probably not worth figuring out a second file format just for that.)

Yes, they are of no use really, outside of Pegasus (the product). Also, when you delete a email msg , like many methods with deleting, the data is only flagged as being deleted and has not been physically removed. You need to 'rebuild' the mailbox, which physically removes the dead data and rebuilds the index

I have a number of pmail folders myself and, as I said, have some interest in working on a perl module for this. I am very unlikely to get around to doing it on my own, however.

Possibly you would be interested in working on the contribution, if Mark wants to extend the module ?

As far as a formal specification of the format, I do not believe one has ever existed.

In theory at least, I don't think it is _too_ hard to work out the layout, I checked some of it out (see my other reply)

I suppose that David Harris is using the source code or comments in the source code as his documentation, and he is unwilling to let out the source without an NDA. (This also is why there is no Linux port. David is ammenable in theory to the idea of someone doing such a port, but they would have to sign an NDA and meet other criteria. An unusual attitude for a freeware developer, perhaps, but Pegasus is of unusual quality, as well.

Pegasus is a very good product, especially considering it has been around for so long as freeware. David only gets an 'income' from people who buy the manuals, so I don't know how he keeps 'bread on the table', so to speak. :)

Thanks,

Peter