Battling mail timing issues

oko1 has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, Earthlings! Take me to your... um, never mind, those guys don't speak Perl anyway. So - greetings, fellow Monks! :) I'd appreciate some advice on a problem that I've run into (*no*, it does not involve UFOs; that would be too much fun.) I will confess that it's not specifically a Perl problem - although I am using Perl to code the relevant bits.

I have an email account on a system where the sysadmin does not want to run a POP server (understandable, given the security issues around POP); this means that my access to the mail on that server is restricted to actually logging in and reading it there. Well... kinda. What I actually do is run a script that uses 'ssh' to 'cat' the contents of that remote mailbox to my local system, then pipes the contents into 'formail -s procmail' (which splits the box into individual messages and processes each of them via 'procmail', which in turn filters it for spam, distributes it to the appropriate mailboxes, etc.) Note that has a possible problem built in: if I do this while the remote mailbox is getting updated, I'm going to get a partial message... but this isn't going to result in anything horrible ('procmail' may complain), so I'm willing to live with that possibility.

The real problem comes when I want to "flush" the content of that remote mailbox after getting its contents. I could just execute a command (via 'ssh') to empty out the file - but that would be a really stupid thing to do. What if the remote SMTP server is writing another email to that box just as I flush it? Bad stuff, that's what.

I've read up on various bits of SMTP server operation (Exim, specifically, since that's what's being used on that remote machine.) I've also noodled on various ideas like copying the remote box to a temp file, then checking for a size difference and deleting/emptying as appropriate if they're the same size (nope, still a possible race condition there). I've also wondered if the SMTP server uses any sort of a lock file to write to the mailbox (it doesn't, as far as I can tell.) It would also be Really Nice (obviously) to eliminate that first problem, where reading the box may result in a partial message.

Unfortunately, at this point, I'm stuck - no new ideas about this problem have hit the old brainbox in more than a week. If anyone has any Wisdom(™) to share, I'd be very grateful.

Update: Apologies about the slow response time, folks - I live on a sailboat, and I've spent the last couple of days preparing for Tropical Storm Fay (which is slowly grinding through us right now.) Perl has not been the top issue on my priority list lately.


-- 
Human history becomes more and more a race between education and catastrophe. -- HG Wells

Comment on Battling mail timing issues

Replies are listed 'Best First'.
Re: Battling mail timing issues by jhourcle (Prior) on Aug 18, 2008 at 01:52 UTC
I'd probably stick with using existing mail programs, and automating them using Expect. The basic unix program `mail`, although a bit antiquated should do most of what you'd need, I would think. You might also look to see if there's any provisions for dealing with this sort of situation in fetchmail It's best not to try to look into locking details of the MDA, as it might change in later versions, or the sysadmin might change programs entirely.	[reply] [d/l]
Re^2: Battling mail timing issues by oko1 (Deacon) on Aug 21, 2008 at 19:54 UTC
[blink] Now I feel really dumb - I've used Expect, etc. in the past for automating remote interactions, and I don't know why I didn't think of it this time. Thank you - that's probably the best idea overall, although I can't think, off the top of my head, how I'd automate deleting all messages in the inbox. I'll figure something out, though! -- Human history becomes more and more a race between education and catastrophe. -- HG Wells	[reply]
Re: Battling mail timing issues by jethro (Monsignor) on Aug 18, 2008 at 01:45 UTC
About the security issues of POP: Are we talking about the issue of password-sniffing ? That can be avoided with the help of ssl or challenge-response. Or is it those old pop servers seem to be full of security holes? Then maybe a newer imap (and pop) server like dovecot might be the solution. At least I have the impression that the dovecot teams promise of high security has not been refuted yet. Exim has to do some sort of file locking. How else could an email reader access the mbox file and for example delete a message? A quick search on the internet produced this page about a utility that uses the locking mechanisms of exim: http://www.exim-new-users.co.uk/content/view/89/39/ . Don't know if it comes with exim or has to be installed separately. It also says that fcntl() (i.e. using the kernel/filesystem based locking) is the default locking mechanism of exim. But a look into the config file of exim should give some indication what locking mechanism is really used.	[reply]
Re: Battling mail timing issues by Krambambuli (Curate) on Aug 18, 2008 at 07:33 UTC
...I have an email account on a system where the sysadmin does not want to run a POP server (understandable, given the security issues around POP); this means that my access to the mail on that server is restricted to actually logging in and reading it there. ... How exactly do you do this? That's the starting point to solve your problem, I think - build your way of managing mail around the mailbox access methods that the remote sysadmin offers and maintains. To second an already expressed opinion - exim+dovecot is a fine solution, we are offering some hosting based on such a combination and it works well for years already. If you have shell and ssh access to the remote server - (a far more dangerous option than running a POPS enabled mailserver, if you ask me) - it might be that forwarding X with ssh's -X option (if you're on Linux) and/or tunneling a local port to a remote one that you are allowed to access (with something like -L local_port:localhost:remote_port) would lead to a convenient solution too. Krambambuli ---	[reply]
Re^2: Battling mail timing issues by oko1 (Deacon) on Aug 21, 2008 at 20:06 UTC
exim+dovecot is a fine solution [...] I'm sure that it is - but it's not available to me, since I don't run that system and the administrator has already said that he doesn't want to install a mail retrieval method beyond what's currently available. 'ssh -X' forwarding isn't necessary either, since I don't use GUI mail clients; in fact, SSH forwarding is not a solution at all, since it doesn't address my problem (synchronizing files, etc., rather than accessing a service.) Thank you for your suggestions, in any case. -- Human history becomes more and more a race between education and catastrophe. -- HG Wells	[reply]
Re^3: Battling mail timing issues by Krambambuli (Curate) on Aug 22, 2008 at 12:16 UTC
... he doesn't want to install a mail retrieval method beyond what's currently available. But what is available ? Is there nothing than direct file access through shell utilities to the Inbox ? If so, ok, I haven't said nothing :). But otherwise ssh -L should solve your problem with ease and confort. Krambambuli --- occasionally enjoying Mark Jason Dominus's Higher-Order Perl	[reply]
Re: Battling mail timing issues by NiJo (Friar) on Aug 18, 2008 at 19:56 UTC
`echo "you@local.box" > ~/.forward` gets rid of all your problems. Mails arrive on the "local" box one by one and are not stored on the "foreign" server.	[reply] [d/l]
Re^2: Battling mail timing issues by oko1 (Deacon) on Aug 21, 2008 at 20:15 UTC
Unfortunately, it does not resolve my problems - or, more accurately, does not result in a configuration that I want. Perhaps I should have mentioned that I don't want to lose the option of reading my mail remotely; there are times when I'm away from my laptop and have temporary access via someone else's machine, and at those times, reading my mail at that remote host is just what I want to do. However, when I am using my laptop, I'd like for the mail from that remote host to be retrieved automatically - a cron job would do that for me if I had a script that would reliably do a retrieve-and-delete of the kind that I already described, without unintended loss. I believe that jhourcle has nailed it: 'Expect' should give me pretty much everything I need - perhaps with 'mailx' on the far end to do the actual message/locking/etc. handling. -- Human history becomes more and more a race between education and catastrophe. -- HG Wells	[reply]