parkan has asked for the wisdom of the Perl Monks concerning the following question:

I need to run background tasks at arbitrary points in the future, with queue behavior if multiple execute at or near the same time, ability to manipulate the task list, retry (ideally with exponentially longer retry window), etc.

In Ruby, there is delayed_job (and php port djjob), which does this pretty much perfectly.

The closest Perl solution I could find was Beanstalk::Job, but this doesn't appear to support execution in the "far future" (next week). There is also Schedule::At, which does (and also allows manipulation of the tasks) but it runs external commands and the jobs appear to be executed straight with no queueing mechanism

Is there something that's more like delayed_job? Do I need to roll my own? Also: beanstalkd appears to be memory resident, mostly, and I'd prefer to have a db/disk backed queue

Replies are listed 'Best First'.
Re: Equivalent for delayed_job?
by Zzenmonk (Sexton) on May 24, 2013 at 07:32 UTC

    Hi Parkan,

    I am asking myself if you have chosen the right solution here. A friend of mine makes his living with "this". What you are trying to do is called a batch management system. There are commercial solutions for this. For example Control-M or the Oracle Batch architecture: http://docs.oracle.com/cd/E17904_01/bi.1111/e16630/batchjobs_framework.htm#autoId0. Notice: these are high end solution, but I know light framework exist also.

    With respect to the complexity of this area ( my Friend is a good German Engineer!), I would have a look at these solutions rather than try to develop such a system. The issue you will face in this area of IT are extremely complex.

    Depending on your requirements, you could think of an Oracle back end with a PERL web-application front end or something similar.

    K

    The best medicine against depression is a cold beer!
[reply]
      Do you know of a specific example of a lightweight system for this? I certainly don't want to reinvent the wheel, but delayed_job comes pretty close to doing everything I would want to "reinvent". The trouble, of course, is that we have a LOT of perl code that performs the actual tasks, and I don't want to just exec() the scripts and divine results from exit codes. It may be helpful to know that the scheduling of the tasks has less to do with available resources and more with the changing state of the outside world across time (we're checking status of various external resources which may sporadically, or sometimes predictably, become unavailable)
[reply]

        Hi

        I could not get in touch with my friend so I looked a bit around. There are tons of them with more or less features. The choice depends on exactly what you want to do with it. As I said I only experienced the high end ones. I guess the best is you google with batch management software. There is one company which seem to have a good product: Skybot.

        Regards

        K

        The best medicine against depression is a cold beer!
[reply]
Re: Equivalent for delayed_job?
by vsespb (Chaplain) on May 24, 2013 at 11:03 UTC
    Did you look at Gearman ?
    It's initially written in Perl, then rewrited. But I believe it should have good Perl API. And I know that it's used on production with Perl.
[reply]
      Gearman looks like a great solution to dispatch and queueing, and parallelization/cross-language calls are great bonuses, but I don't see a way to schedule jobs for specific future times. Am I missing something?
[reply]

        I did not check if this particular feature exists. I just suggested that you can take a look on Gearman as it's pretty good solution for perl..

        If feature does not exists, maybe you should change your workflow a bit.. Maybe CRON + Job server ?

        I have experience working with several project (using Ruby on Rails+delayed_job and Ruby on Rails+resque) where scheduling done in such way:

        Cron (or analog) task executed at particular point of time. Task is lightweight and what it do is scheduling of (a thousands) of tasks to be immediately executed by job server.

        For example if you sending massive weekly newsletter at 19:00 Saturday. That's just a cron task which schedule it. Actual send is made by job server

        If you need to expire user passwords in 30 days after it last changed - it's a task, ran each day, which fetches expired passwords and queue expiration

        If you need expire sessions after 1 hour, it's a task again, which is run each minute, it fetches expired sessions and queue expiration tasks for each session

        Or you have use-case which does not fit this?

        p.s. in Rails we also had to use some framework to autogenerate cron tasks at deploy time

[reply]
Re: Equivalent for delayed_job?
by rahulruns (Scribe) on May 24, 2013 at 04:45 UTC

    I assume your environment is LINUX(UNIX Based) and if my assumption is good why not to write the PERL SCRIPT and use a cron job. It would be a bit easier I believe.

[reply]
      Cron cannot compete with job servers.. It's a bit different thing.
[reply]
        Correct, the granularity and complexity of managing these tasks as cron jobs would quickly get out of hand.
[reply]
        But I believe cron can be used in conjunction with job server http://www.perlmonks.org/?node_id=1035259
[reply]

Back to Seekers of Perl Wisdom