madbombX has asked for the wisdom of the Perl Monks concerning the following question:

Hello all,

I am looking to download all messages from an IMAP server (Exchange) from two folders to two files in mbox format. I have done this using Mail::IMAPClient; more specifically in the following manner: 

 foreach my $mbox qw( SPAM HAM ) {
   $imap->select($mbox);
   my @msgs = $imap->search("ALL");
$imap->message_to_file("/home/mail/mbox/$mbox".".msgs",@msgs);
   $imap->delete_message(@msgs);
   $imap->close();
 }

[download]
This seems to work just fine except the files HAM.msgs and SPAM.msgs aren't in mbox format, they are just files with lists of messages in them. In an attempt to convert them, I am using the following command: `formail < /home/mail/mbox/HAM.msgs > HAM`. However, that leaves me with an mbox file that just has 1 long messgae according to mbox readers (however that long message contains all the messages in the file).

Therefore my question is threefold:

  1. Would it be better (or work for that matter) if I could somehow be using those pipes in $imap->message_to_file so the messages could be piped directly through formail avoiding an unnecessary step?
  2. Is there a Perl Module that would do this on the fly or do a better job than I appear to be doing it using system commands?
  3. Does someone have a better idea?

Ultimate goal: To download the 2 folders which contain HAM and SPAM messages, feed them through sa-learn (spamassassin which requires mbox format), convert them to Maildir format using mb2md (which I already have done) so I can have dspam_train learn them (because dspam requires Maildir format to learn).

UPDATE:

This is the code that eventually ended up doing exactly what I needed it to do. Thanks to everyone for all the help, it is greatly appreciated. I will be putting the final script up on my webpage next week (http://eric.lubow.org/projects), it will be the IMAP SPAM learning script for those interested: 

 use Mail::IMAPClient;
 my $imap = Mail::IMAPClient->new( Server       => 'imap.server.com:14
+3',
                                   User         => 'user',
                                   Password     => 'pass')
        or die "IMAP Failure: $@";

 foreach my $box qw( SPAM HAM ) {
   my $file = "/home/mail/mbox/". $box;
   $imap->select($box);

   my @msgs = $imap->search('ALL')
        or die "Couldn't get all messages\n";

   foreach my $msg (@msgs) {
     open my $pipe, "| formail >> $file"
       or die("Formail Open Pipe Error: $!");
     $imap->message_to_file($pipe, $msg);
     close $pipe
       or die("Formail Close Pipe Error: $!");
     $imap->delete_message($msg);
   }

   # Now expunge the messages and close the folder
   $imap->expunge($box);
   $imap->close($box);
 }

 $imap->logout();

[download]

Replies are listed 'Best First'.
Re: Downloading IMAP Folders to mbox file
by sgifford (Prior) on Jun 17, 2006 at 21:04 UTC
[reply]
      I don't see how this can work in conjunction with Mail::IMAPClient's message_to_file since message_to_file's messages already have message id's which, according to Mail::Box::Mbox, they will then get passed over. And also, I wouldn't be able to pass it a message object like it wants, I would only be able to pass it (at best) a message individually from the IMAP download. This also doesn't seem to have the ability to repair an mbox mailbox which is what seems like it could help.

      Eric

[reply]
        You should be able to construct a Mail::Box::Message object from the IMAP data. Alternatively, it looks like Mail::Box::IMAP4 might help you get messages in the right format easily.

        Update: Here's an example: 

        #!/usr/bin/perl

use warnings;
use strict;

use Mail::Box::Manager;

my $mgr = Mail::Box::Manager->new
    or die "Couldn't create manager\n";

my $imap_client = Mail::IMAPClient->new(Server => 'suspectclass.com',
                    User => 'username',
                    Password => 'pass')
    or die "Couldn't create IMAP client\n";

my $imap = $mgr->open(type => 'imap',
               imap_client => $imap_client,
               folder => 'INBOX')
    or die "Couldn't create IMAP client\n";

open(M,'>> INBOX.mbox')
    or die "Couldn't create INBOX.mbox: $!\n";
close(M);
my $mbox = $mgr->open(type => 'mbox',
              folder => './INBOX.mbox',
              access => 'rw')
    or die "Couldn't open MBOX folder\n";

my @m = $imap->messages('ALL')
    or die "Couldn't get all messages\n";

$mgr->copyMessage($mbox,@m)
    or die "Couldn't copy messages\n";

        [download]

        --
        sgifford's Web page
[reply]
[d/l]
Re: Downloading IMAP Folders to mbox file
by TilRMan (Friar) on Jun 19, 2006 at 06:17 UTC

    You probably need to supply some arguments to formail -- perhaps some of -Yeds -- to help it figure out where your messages start and stop. However, formail can't promise to get it 100% right, so you'll end up with mutilated and merged messages.

    If you were as paranoid as I am, you would run each message through formail and concatenate the output into a single file; that file would be a valid mbox. This will take a long time for many messages; but if you are running this regularly it should be tolerable. 

    # Untested
foreach my $flavor qw( SPAM HAM ) {
  my $file = "/home/mail/mbox/${flavor}.mbox";
  $imap->select($flavor);
  my @msgs = $imap->search("ALL");
  foreach my $msg (@msgs) {
    open my $pipe, "| formail >> $file" or die "$!";
    $imap->message_to_file($pipe, $msg);
    close $pipe or die "$!";
    $imap->delete_message($msg);
  }
}

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom