Seriously.

Keep a small table of open jobs in a database table. For each step open a transaction, select a job, and attempt to do one step. If you succeed update the table with the new information about what comes next and commit the transaction. If you fail send email and rollback the transaction. Should you crash unexpectedly, it is up to the database to ensure that there is a reliable record of your current state.

Just be sure that each of your labelled steps is one which from the time you open a transaction to the time you commit it, can be restarted from scratch without penalty.

Should you wish to create a log of your actions, that log can be kept in the database as well.

I think two things about my question are implicit that I will now make explicit:
• I like the database idea. The only catch is state preservation. In a language like Lisp, where data and function are interchangeable and all data and function has a unique printed representation, executable "codelets" can be written and positions and the "codelets" stores in a tilly-esque transaction. In perl, code blocks do not have a printed and re-executable representation unless you break down to string manipulations to create and eval your code, which is highly unstructured and error-prone.
• What we have hear is what Douglas Hofstadter defined as GOD in "Goedel, Escher, Bach, the Eternal Golden Braid." GOD == God Over Djinn. So each god has a higher God ad infinitum. Once you have the database serving as a god for the simple perl code, you have to ask who is the God for the database code? And then God for that?

  And then we conclude that linear thinking leads to the unreal, impossible, and irrational concept known as infinity --- scientfic man's attempt to justify a day-to-day practical form of reasoning (linear) when cyclical reasoning is actually a bit more suited to reasoning about Universal truths.

I would make state an enumerated type. There is no need to have the database know the details of how a step is to happen or what language it is implemented in. Conversely there are significant security issues in trusting such information from a database machine. Should your database be spoofed or compromised, it is better to just have your process fail rather than giving the attacker a chance to compromise your machine as well!

The God for the database code is the fact that it is designed to properly log all transactions and be in a position where at all times it is in a consistent state on disk. Even if there is hardware failure. And if you have a hot backup, you may even be protected from that.

Getting failover, etc right is a hard problem. Databases already solve it, so leverage off of what they do rather than rolling your own.

Note that this basic transaction-oriented strategy scales to very complex problems, and gives you systems that are robust by default. (One I dealt with got nicknamed "accidentally robust" because of how many unexpected error conditions came up, had never been considered, and didn't cause serious problems.) Normally when you rely on certain things having just worked and others being about to, when something goes wrong (eg a done file doesn't get written) things careen off-course into a nightmare. But with this strategy you automatically stop at a point where minimal damage has happened, other parts of the process can continue, and when the underlying problem is fixed, you are generally in an easily recoverable state...

-- Randal L. Schwartz, Perl hacker

If there's a failure, you're relying on e-mail to deliver your failure notification... e-mail isn't exactly the most reliable mechanism out there.

You are talking about a stage queue. Something like journalling filesystems do. Stage Queuing needs to be tuned to the individual app. I did this once for signup data at an ISP. The CGI wrote a file in a queue directory and then tried to go send the request. If it succeeded, it wiped the file. A cron job swept thru every minute and looked for files who were older than 5 minutes, killed their cgi and tried it itself. Any file older than 1 minute with no cgi got tried too.

It's a royal pain in the butt and you have to do all kinds of forward and back checks to make sure you don't flood yourself with duplicates, lose important data and more. Worse, your front line interaction system needs to understand all of that and sniff what stage it's at if the user disconnects and comes back.

And after all that; it maybe fixed two customers automatically in 6 months. You need this almost never. =) It better be real improtant or it just ain't worth it.

--
$you = new YOU;
honk() if $you->love(perl)