When faced with anything that looks like a pipeline process, I like to split up the parts into many small parts, and keep each one as simple as possible.

It seems to me that you have three steps:

1. Procure the incoming data
2. Post the data on the external forum
3. Put the data into the processing queue for the next process

As long as performance permits it, I would use only one process for one step, as multiple processes will give you the headaches of concurrency.

If you have a scheme of proper file locking (as it is easily available under Win32, and not-so-easily-but already demonstrated here under Unixish filesystems), you can use a separate process for each step, which makes restarting certain items much easier. Then, each item becomes a file which is moved from directory to directory as it progresses through the stages of your pipeline. Status queries are then reduced to the task of finding where a file resides in the directory tree and to checks that no file should be older than (say) 5 minutes in any of the intermittent directories.

If you have no way of proper locking through files, a database supplies you with easy concurrency and proper locking. Put your data in a table row, together with a status column and all processes can even easier manipulate the data. I would still restrict input to one process to avoid feeding duplicates, but if you construct your SQL and the status properly, you can have as many processing items as you wish/your system allows. Status queries are then simple SQL, but taking an item out of the processing pipeline requires setting the status instead of moving the file - this may or may not be a problem for bulk changes, depending on how much access you have to the live system.

Corion,
I am responsible for only one piece in the pipeline. The piece I described. The process that will be filling my work queue and the process whose work queue I will be filling is beyond my control. I do have a bit of say as to how my work queue gets filled just as I have to conform to the requirements of the work queue I will be filling.

My preliminary tests and estimated rate of incoming jobs leads me to believe single threaded serial processing will be insufficient. That isn't to say I have missed something where I could improve efficiency, but I really do want to have my options open.

I like the simplicity of 1 file per job and using directories as queues. I am not sure if there will be a "reporting" requirement in the future, but I can bet if I don't plan for it there will be. That leaves a couple of options as I see it:
• Write a transaction log
• Use a database where the stats can be constructed (time in, time out, number of attempts, etc)
Any code snippets will be appreciated. Thanks again.
L~R

Aside from the other suggestions, might I recommend a serializable workflow object? You would basically draw a work flowchart with a beginning and an end. Each stage is a different "state" in the workflow with a proper commit/rollback procedure. I think Pixie might be great for something like this. Your program could load all workflow objects and either pass each object to a different thread/POE process or just load them sequentially if the load was light enough.

For the "post to form" job, the workflow object would attemp to post the data and, if failing, move itself to the bad queue. For that, you might want a single workflow collection object. You only need (hopefully) one instance and use the flyweight pattern to identify the workflow objects in various "in", "work", "out", and "bad" queues. (I'm just guessing about the flyweight pattern because I think it might make it easier if you're running multiple threads or have workflow objects on another marchine).

Conceptually I think it's a pretty clean model and hopefully would accurately reflect your business needs. It also makes locking simple if you have a single collection object assigning tasks to the workflow objects (theoretically).

A way ensure that, even if the database doesn't allow atomic operations is to have a field (call it lock (int)) in your row. When you want to select that row, do a

update table set lock=$$ where id=foo and lock=0
[download]

/var/spool/prog/queue-1
/var/spool/prog/queue-2
/var/spool/prog/queue-bad
/var/spool/prog/queue-lowpriority
[download]

I'm glad to have found this discussion as I am facing a similar issue. It seems the consensus for managing work-in-progress is via a status field in the work queue database. That's the approach I was considering so am glad to see it validated.

However, none of the posters who suggested this method gave any input about how to recover from error conditions. How does the script know if a record marked as 'in-progress' is actually being worked, is stalled (because the user took a lunch break), or needs to be returned to the queue because of a system crash, user error or other anomaly?

In my case, each user has an id which I plan to use for checking if there is already a record being worked by this user. It won't catch stalled records if the user doesn't return which is not ideal. I could get around that by adding an additional expiration check for nn minutes/hours/days. Any other suggestions/comments would be most appreciated.

Thanks,
William

knowmad,
I ended up using Parallel::ForkManager. It offers a few callbacks that can hook to custom code to verify this. Since this is a application specific type of verifiecation, I think the way it was abstracted in the module was perfect for my needs.

Hope this helps - L~R

L ~ R,

Thanks for your reply. I looked into Parallel::ForkManager but don't really see how you are using it to manage a work queue for a web application. Do you keep a forked process open until the user submits the form? Which callbacks are you using?

Thanks for your feedback,
William