Neid has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm building a tool with parent-workers architecture. The parent will wait for jobs
using HTTP server and dispatch tasks to workers. Each worker is a separate process.
There also will be client which will send jobs to parent or parents ( there will be many servers).
Each parent should be able to ping workers from time to time, and get task status information.

For the whole architecture I would like to use POE.
But I'm POE novice.

For the client I'm using:
POE::Component::Client::HTTP

For the parent-workers I will probably use:
POE::Wheel::Run

But I don't know what to use for parent-workers communication and task assignment.
Is POE::Component::JobQueue appropriate for task assignment?

And what is the best for communication?

Thanks for advice,
Neid

Replies are listed 'Best First'.
Re: POE modules advice
by rcaputo (Chaplain) on Mar 19, 2009 at 20:23 UTC

    It sounds like you're putting the cart before the horse. POE::Wheel::Run includes parent/child process communication. POE::Component::Client::HTTP is not a a HTTP server. If your "clients" are simple programs, they could be implemented with plain LWP to talk to a POE-based HTTP server. You haven't explicitly said the httpd is running POE; I'm assuming yes because of the POE::Wheel::Run.

    What I think (which is based on incomplete information and is probably wrong):

    • Plain old clients. LPW, curl, wget, whatever. This handles the client/httpd communication.
    • HTTP server, using one of the POE httpd components (your choice).
    • POE::Wheel::Run to run child processes and deal with httpd/child communication.
    • Worker pool management is a vague concept and probably includes some custom logic. Until I know more, you probably write this part.

    I would have the child processes report their status as it changes, and then store that status in the HTTP session (not a POE session). When the client polls for status, it can be satisfied by the session information rather than asking the child process (which may have exited or crashed anyway).

    http://poe.perl.org/?POE_Cookbook/Job_Server is a telnet based job server. It's not appropriate for your task, but it does show one way to write a server that accepts jobs and then runs them in separate processes.

    Sorry for the vague answer. I may be able to narrow down my advice if you provide more specifics.

      Yes that's true, httpd will be also POE (e.g. POE::Component::Server::HTTP ) I saw that POE::Wheel::Run uses stdin/stdout for communication.
      My workers is a perl code so I would pass subroutine reference to Wheel::Run 'Program' argument.
      Unfortunately that code may print some debug stuff.

      I was thinking about worker code as something similar to:

      sub worker{ my $self = shift; while( my $task = $some_queue->dequeue ) { # do_something() can print debug output to STDOUT my $result = $self->do_something(); my $status = { result => $result, time => time(), worker_id => + $self->id}; $self->update_status( $status); } }

      And parent code something like:
      sub parent { my $self = shift; $some_queue->enqueue( $new_task); } sub update_status_event{ my $status = shift; # do something with status #... }

      So I think POE::Wheel::Run for inter-process communication is not enough.
      Any advice here?

      Thanks,