This sounds like a job for a queue - something like RabbitMQ. It would give you the persistence and allow you to scale even farther out if necessary across multiple servers or even instances of your processes.

I would also consider using a NoSQL db such as CouchDB to hold your data if passing your data via the message body in the queue is not enough.