for  my $file (<*.imap>)

I believe this is going to build a list of every file you want to operate on, and then iterate over it. If that list is very long, you're going to spend a lot of memory storing it. The code below does the same thing, but it reads one directory entry at a time.

opendir my $cwd_dh, '.'
    or die "Can't opendir: $!";
while ( my $file = readdir $cwd_dh ) {
    next unless $file =~ m{ \. imap \z }xms;
    # ...
}
closedir $cwd_dh;
[download]

I would not expect this to be faster unless the memory used by the one big glob is sending you into swap.

Have a look at opendir and readdir for details about this method.