dgaramond2 has asked for the wisdom of the Perl Monks concerning the following question:

After shopping around in CPAN, I found that Proc::Reliable provides most of the functionalities that I need for my web application.

The requirements are: web users can submit tasks. Tasks are processes that can take quite some time to finish (e.g. "wget http://host/some_big_file" or "tar xfz some_huge_tarball"). So the web app will queue these tasks in a single big Task Queue instead of running them directly after form submission. There's a maximum number of items for the Task Queue, as well as per-user max. The max values are adjustable depending on system load and other factors. For each task there's a time limit imposed as well as other options like number of retries when failed (which will be taken care of by Proc::Reliable), "nice" value, etc. Web users can view the output and status of his/her tasks, and perform other options like cancelling them or moving the priority of his/her queued tasks.

I'm wondering if there are other CPAN modules which are closer to my needs than Proc::Reliable, or if there are some other complementary modules I can use for less reinvention of wheels.

Replies are listed 'Best First'.
Re: Using Proc::Reliable in CGI
by almut (Canon) on Apr 14, 2007 at 14:07 UTC

    As no one has replied so far, I'll risk to suggest something which might not fit exactly what you're looking for... (after all, you specifically asked for CPAN modules)

    Anyhow, to me it sounds like you might benefit from using an existing job queuing system for the backend, and then only develop some web frontend to your liking. There are commercial solutions like LSF1, but I suspect you might be more interested in open source software, so you might want to take a look at Sun Grid Engine, which provides a similar feature set.

    Both systems might seem like overkill for what you appear to be in need of. On the other hand, no one forces you to deploy a world wide grid of computing resources. You can just as well configure a mini "cluster" of, let's say, 1-3 machines, and still take advantage of the extremely flexible queuing, load balancing, resource management and monitoring facilities etc., that such tools provide.

    Before you approach the docs, however, it might help to install a filter (in your head) to mask out all the enterprise level blurb you're likely to encounter ;)

    There also is a Perl binding to talk to the queuing backend via DRMAA (Distributed Resource Managment Application API), but I cannot say much more than this, as I've never used the module myself.

    Hope this helps, and good luck!
    Almut

    ___

    1  Disclaimer: I'm in no way affiliated with Platform Computing Inc., the producer of LSF. I just happen to have had some exposure to the software at work.

[reply]
      Thanks for the reply. Initially I followed the link just to see if the software contains Java so I can have a "rational" basis to reject it. But it doesn't, and is in fact pretty interesting after all. I don't think I'll be using it yet for my current project, but will play with it a bit sometime soon. Too bad the buzzword "grid computing" is passe nowadays.
[reply]
Re: Using Proc::Reliable in CGI
by aufflick (Deacon) on Apr 15, 2007 at 08:47 UTC
    Wow - just the module I've wanted for a while - thanks for posting!
[reply]

Back to Seekers of Perl Wisdom