Kallisti has asked for the wisdom of the Perl Monks concerning the following question:

Hello, time for my first question. ;)

I want to write a daemon in perl that receives new events from web scripts via POE::Component::Server::IKC and manages a queue of events that will be executed at a special point of time (selfwritten using POE too).

Currently, I implemented POE::Component::JobQueue to work on the events that are due for processing.

But since the events might take longer to process and should not block other parts of the program, I would like to rather use a pool of forked processes instead of cooperative multitasking.

Now the question is how can I realize the worker pool with the least work and trouble.

What I need is:

I searched CPAN for a while (disregarding modules in version 0.0001 / with bad documentation) and found the following modules:

POE::Component::Pool::Thread This one uses threading, which is, according to all i've read, not the way to go in perl on *nix, if it is possible. So I would rather not use it.

POE::Component::PreforkDispatch This one does exactly what I want. It lacks the "babysitting" and process watching stuff, but I think I could implement that using POE::Component::Supervisor, later on. I still wonder if it is stable enough.

I just wonder if XMLRPC isn't a bit too much for just sharing a hash of data...

POE::Component::Daemon I did not try this one out explicitely, but according to the description, it looks promising too. The "babysitting" is "probably badly implemented" according to the documentation... Using this module seems to be a bit messy, if I got the stuff with SocketFactory and status updates right in the documentation.. just doesn't look as high level and funny as the PreforkDispatch one...

All three modules are still in version 0.1* which makes me a bit careful. I would like to use proven / stable code. So if you have any experiences, I would be glad to hear of them.

Should I write it from scratch or can I safely use one of the modules above? Do you have other recommendations for me? I'm happy about all comments, thanks in advance.

  • Comment on POE, daemon, how to realize pre-forking worker pool for event processing?

Replies are listed 'Best First'.
Re: POE, daemon, how to realize pre-forking worker pool for event processing?
by plobsing (Friar) on Sep 29, 2008 at 04:10 UTC
    Use one of the CPAN solutions first. If you don't trust them, read the source. If they don't work for you, patch them.

    Without users, how do you expect a library to go beyond 0.1?

    Consider the alternative: You write a library yourself. What version should it start as? 0.1 of course! How is that any better?
Re: POE, daemon, how to realize pre-forking worker pool for event processing?
by waba (Monk) on Sep 30, 2008 at 17:43 UTC

    I implemented a similar system at $work using PoCo::PreforkDispatch. It works pretty well and does replace crashed workers. If you are concerned about frozen jobs, you can easily avoid this problem using a classic alarm signal.

    We're having just one issue with this setup however, sometimes the controlling process gets completely stuck on something and has to be killed. I don't know yet if PreforkDispatch is the culprit, but I doubt so. I can keep you posted if you want. Update: this is a MySQL issue, POE and its components have nothing to do with that.

    (updated) As for father-children task/result IPC, there is no need for XMLRPC. I think this piece of code is a left-over of an earlier design. Actually, PoCo::PreforkDispatch leverages POE::Filter::Reference, which will serialize using Storable the arguments and return values of your function.

      I had the exact same problem with something in MySQL hanging POE::Component::PreforkDispatch. What did you have to fix?