Parent/Child,Boss/Worker->IPC suggestions

josh803316 has asked for the wisdom of the Perl Monks concerning the following question:

Problem:
I have a worker daemon (gearman) that will be acting as an automated test manager. The manager will receive client requests to start a task or group of tasks and will send those requests to a pool(queue) of workers.

I would like (during app startup) to fork a pool of workers that will be doing long running testing tasks but will be able to communicate with the manager so that the task may be stopped, paused, re-started, status may be queried, etc....

I'm looking for suggestions on which modules might help me best achieve my goal....or just a general guide on the best way to proceed with this type of application design. I have looked at Parallel::ForkManager as well as POE::Component::PreforkDispatch along with Parallel::Fork::BossWorkerAsync but am not sure if these are right for me or if there are more powereful ones out there.

Just looking for some guidance or thoughts on the matter...

Comment on Parent/Child,Boss/Worker->IPC suggestions

Replies are listed 'Best First'.
Re: Parent/Child,Boss/Worker->IPC suggestions by perrin (Chancellor) on Jun 30, 2009 at 04:47 UTC
I had good luck using Parallel::ForkManager and keeping shared state in a SQL database. I considered many other designs because the idea of polling the db bothered me, but ultimately this one was simpler to code and more scalable than anything else I came up with. However... Isn't gearman supposed to do this for you? Why don't you make the jobs that you want to pass off to the workers into gearman jobs?	[reply]
Re^2: Parent/Child,Boss/Worker->IPC suggestions by josh803316 (Beadle) on Jun 30, 2009 at 07:34 UTC
Yes, my original design was to have gearman do this for me but the problem I've been facing is shared memory, communication between parent and child workers and blocking during a long running gearman worker process.... I wasn't sure how to create a manager/worker setup, all on the gearman worker side. I guess I could have the manager be a gearman worker and a gearman client at the same time. (Thoughts??).	[reply]
Re^3: Parent/Child,Boss/Worker->IPC suggestions by perrin (Chancellor) on Jun 30, 2009 at 15:14 UTC
Yeah, I advise you not to use shared memory. I'd keep state in a database, and use signals if you need to interrupt running processes to shut them down.	[reply]
Re^2: Parent/Child,Boss/Worker->IPC suggestions by metaperl (Curate) on Jun 30, 2009 at 14:05 UTC
I second shared state in a SQL(ite)? database. I tried a ton of `IPC::*` modules and couldnt get any of them to work. I also tried <CODE>Cache</CODE to no avail.	[reply] [d/l]
Re: Parent/Child,Boss/Worker->IPC suggestions by tmaly (Monk) on Jun 30, 2009 at 14:39 UTC
I have used POE::Component::Generic to help with the children. Just wrap your objects with this an the POE framework can handle the spawning and control.	[reply]
Re^2: Parent/Child,Boss/Worker->IPC suggestions by josh803316 (Beadle) on Jun 30, 2009 at 15:47 UTC
Thanks very much for this suggestion, this might be the way to go for me to have more control or interactive power over the long running test.	[reply]