Have you looked at Mail::Thread?

$hash{$msgid} = {
    filepos => $msgstart,
    inreply => $msgreply,
    nextmsg => []  # reference to an empty array
}
[download]

my @heads;
while(my ($key, $value) = each(%hash)) {
    # does this msg have a parent?
    if (my $parent = %hash{$value->{inreply}}) {
        # link the parent to this message
        push @{$parent->{nextmsg}}, $key;
    }
    else {
        # no parent so it must become a thread start
        push @heads, $key;
    }
}
[download]

foreach my $msgid (@heads) {
    # do what ever you have to do to start a new thread
   
    # recursively follow the messages
    thread_msgs($hash{$msgid});
}

# recursively read through the thread
sub thread_msgs {
    my $msgid = shift;

    # use seek() to position the mbox file at
    # the message start using $msgid->{filepos}
    # copy to the end of the message

    foreach $msgid (@{$msgid->{nextmsg}}) {
        thread_msgs($msgid);
    }
}
[download]

Once you have this hash, you can then (relatively) quickly identify which messages go with which heads:

• $hash{A} is null (or a message-id which isn't in the hash), so it's a thread head
  
• $hash{B} is A, and $hash{A} is a head, so it's in A's thread
  
• $hash{C} is B, but B isn't a head, so look at $hash{B}, which is A; A is a head, so C is in A's thread
  
• $hash{D} is C, but C isn't a head, so...

Once you've identified the head of the thread that each message is in, you can then build the hash you originally attempted, mapping each head to an array of messages in that thread.

I was going to suggest to keep track of message IDs in a hash with each reply being a value for the parent message. To figure out the "Message-ID:" & "In-Reply-To:" values to extract, i saw " $obj->threads([FOLDERS], OPTIONS)" in "Mail::Box::Manager" pod.

So, try the following & see if it makes any difference ...

#!perl

use warnings; use strict;

use Mail::Box::Manager;

my $mgr = Mail::Box::Manager->new( 'default_folder_type' => 'mbox' ) ;

foreach my $mb ( @ARGV )
{
  my $mbox = $mgr->open( 'folder' => $mb )
    or do { warn "Can't open $mb"; next; };

  my $threads =
    $mgr->threads
    ( 'folders' => [ $mbox ] , 'timespan' => 'EVER' , 'window' => 'ALL
+' );

  save_thread( $_ ) for $threads->all ;

  $mgr->close( $mbox );
}

{
  my ( $count , @stat );
  sub save_thread
  {
    my ( $thread ) = @_;

    #  Generate file name.
    my $file = 'thread-' . sprintf '%05d' , ++$count;

    my $save =
      $mgr->open( 'folder' => $file , 'access' => 'rw' , 'create' => 1
+)
      or die "Cannot open $file to save the thread.\n" ;

    push @stat , [ $count , $thread->numberOfMessages ];

    $_->copyTo( $save ) for $thread->threadMessages ;

    $mgr->close( $save );
  }

  sub END { print_stat(); }

  sub print_stat
  {
    my $out = '';
    my ( $total_thr , $total_msg ) = ( 0 ) x2;
    foreach my $s ( @stat )
    {
      $out .= sprintf "%4d : %2d\n" , @{ $s };
      $total_thr++;
      $total_msg += $s->[1];
    }

    my $old = select STDERR;

    printf "Threads: %4d,  Messages: %4d\n%s\n"
      , $total_thr , $total_msg , join '' , qw( =- ) x20 ;

    print $out;

    select $old;
  }
}
[download]

For my test case, mind that above code generates triplicates in some cases (but "mutt(1)" does not have the problem in generating threads). In addition, while "mutt" notices 602 threads, above code produces 675 (but that could be /in some part/ due to various threading options that are set for "mutt"). I can provide the original file and those created by "Mail::Box*" if anybody is interested.

*Update, May 23 2006* Added a simple sub to print the statistics; removed unused "Mail::Message" usage.

Thanks for the help guys - much appreciated :) - I wasn't expecting such a collection of comprehensive and speedy responses ;)

Fletch - I hadn't seen that module before - however I had read the article regarding threading that it links to. Don't know how I managed to miss the module... Thanks for pointing it out

ruzam - Thanks for the code - I had a quick play with it, and couldn't get it working as I wanted. I'll have another crack at it, as I have only looked at it briefly.

esper - Sorry, should have been clearer in my original posting - I did create a hash, where the keys were every message-id and tried your first suggestion - but it was indeed too slow. I didn't think of creating a 2nd hash containing the thread's head. I'll give that a shot, sounds like it could work!

roboticus - Unfortunately, Yahoo groups threaded view goes by subject rather than message-id - giving me some really weird threads as there are thousands of messages with the subject "hi" in my mbox ;) - Also, it took me 3 weeks to get the messages downloaded, as yahoo prevent you from accessing pages too quickly, eventually banning your IP from the whole of yahoo groups! - I put a delay in my download script to get around this, but I'm not going to set the download going again!

parv - Thanks for the code - I will test it, and see what I can do with it. I've never used mutt before, so it may take a while for me to try you other suggestions. If I can't get it working any other way, then I'll give it a shot.

Yahoo can display groups in threaded view. P'raps you could screen-scrape the hierarchy from that.

--roboticus