The problem is building something like this. Which is why I asked originally if someone knew of any existing opensource, Perl-based effort on this.

Re your suggestion, I don't believe a database is a good option. A database creates another, well, database. No, no. I would envision this app to be merely another view, albeit a value-added view, to the existing mail data store so that the view would always be up-to-date. Otherwise one would have to worry about synchronization. That is the problem with Zoe. Once it has imported everything, it has imported everything, even if it is spam or junk or just plain useless. After that even if one deletes those worthless messages from the mailboxes, they still remain in Zoe.

Otoh, a database might be a useful, additional add-on, but syncing would be very important.

Anyway, thanks for the ideas.

If you want to search things -- dates, senders, URLs, related text, whatever -- quickly, you need to build an index. Looking through 2GB of e-mail on the fly to see what other stuff this person sent will take way too long. You could create your own indexing system, but there's not much point when free databases are available.

It's pretty easy to make your mail system run a script to index each new e-mail you get when it arrives, or to look through your mailbox once an hour and add anything that's new.

You can't do this without some sort of database. At least, not for large mail stores. In order to display a view, you must have a model for it. Searching through every email for every view change would take too long for large mail stores; thus you must index.

That doesn't necessarily mean using an SQL database, though the flexibility would be usefull in this case, IMHO.

To fix your problem, whenever mail is deleted, it needs to be removed from the database. Sounds like it would be good to combine this with a mail client. Otherwise, a periodic re-scan of the mail store is needed to delete old links.

Perhaps storing a hash of each message to ID it and checking to see if it still exists?

There is also the problem of determining what kind of data storage this will look at. Are you wanting to parse mailbox format, Maildir format, or IMAP?

How about a POP3 proxy that indexes all messages as it downloads from the server?

The perltwingular interface could have functions to do deletions and management too, to get rid of spam and junk. Heck, an interface to Spamassassin could help with that.

... I don't believe a database is a good option. A database creates another, well, database. No, no. I would envision this app to be merely another view, albeit a value-added view, to the existing mail data store so that the view would always be up-to-date.

Perhaps you need a paradigm shift. This system, once built and proven reliable, might be regarded as the "authoritative" archive of your email, rather than the old mbox (or whatever) files.

Given a decent interface, most email clients could deal directly with the twingle store rather than the intermediate mailbox.

Matt

Well, such a paradigm shift would come at a great cost, as that would amount to creating a new email client along with its twingled data store.

Right now as is, the email client work has already been done by many worthy agents, in my case the Apple Mail.app.

You are right, however, as this task would require careful thought, particularly in limiting the scope of the project so that it descends from the realm of "would be nice" to "do-able."

Maybe at some time I will become familiar with Java or Cocoa and be able to make sense of the existing opensource apps such as Zoe (Java) and Emila (Cocoa), or better yet, I hope twingle.com would open up their secrets so I could read their Perl source and see what magic is going on.