I second that. You would be increasing your performance by a great deal just by finding a clever way to keep the attachments directly on the filesystem. I would suppose that you have a user primary key and a message primary key, correct? If so, you could try doing a directory tree like:

/users/
      /user1/
            /attachmenta
            /attachmentb
            /attachmentc
      /user2/
            /attachmenta
            /attachmentb
            /attachmentc
[download]

Where userx would be the user pk and attachmentx would be the attachment pk (doh!). On the downside, it would probably be nice to develop a tool or cron job to keep track of lost attachments, old stuff and so forth...

#!/home/bbq/bin/perl
# Trust no1!