Laurielounge has asked for the wisdom of the Perl Monks concerning the following question:

Hi gang

Scheduling. I have need to setup a system to trigger processes at precise-ish times. I have some processes that will run every minute of every day (a number of them - multiple customers, each with a process that,in this case, updates their realtime google analytics data. Some every quarter hour. Every hour. Once a day. It's too many processes for cron to cope with. The every minute ones, for example, I want to stagger over the minute so they're not all firing right bang on the zero seconds mark.

It's not a one time process either - new customer comes on board or customer's needs change and the schedule will have to be re-written. I've done the bit to create the actual schedule (a table with times and the process to be run at that time). What I don't have is the bit - maybe it's a daemon - which checks the time and decides what to run. Or alternatively, decides what the next trigger time is, waits for that time, and fires off the appropriate process, then waits for the next trigger time.

Anyone got any ideas?

Replies are listed 'Best First'.
Re: Scheduling type daemon
by hippo (Archbishop) on Sep 13, 2017 at 11:22 UTC
    Some every quarter hour. Every hour. Once a day. It's too many processes for cron to cope with. The every minute ones, for example, I want to stagger over the minute so they're not all firing right bang on the zero seconds mark.

    So, you need 4 shell (or perl or whatever) scripts like: minute.sh 15min.sh hour.sh day.sh which sit in some directory somewhere. Each script then runs all the jobs to be run on that schedule and does so sequentially (or uses GNU parallel or similar if you want a farm). Then your cron config only needs 4 entries and cron can quite happily cope with that.

    * * * * *    /foo/minute.sh
*/15 * * * * /foo/15min.sh
0 * * * *    /foo/hour.sh
0 0 * * *    /foo/day.sh

    [download]
[reply]
[d/l]
[select]
      The four crons is an acceptable approach but doesn't solve the other bit of the problem.

      Take the every minute job as an example. At stay time either all of the jobs fire in series (or in parallel, doesn't really matter). What I'm trying to do is stretch the load over the period, in the case, the minute.
      As I said, I've got the scheduling database ready. So on the minute when the cron fires, I want to run these few jobs immediately, at the five second mark (say) these ones... for the every 15 minute job, I want to immediately run these jobs, and at the three minute mark (say) run these ones... the five minute (say) run these ones... so the cron job fires a script which then times the execution of the jobs. Which may be timed to the second. The accuracy of the timing is not really important, its just about spreading the load a bit. I say "to the second" as that seems to me the only way to get that granularity in the "every minute" job.
[reply]
        In the script, just run the scripts one by one in a loop. Add a sleep with the value of available seconds / number of jobs. 
        ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

        [download]
[reply]
[d/l]
Re: Scheduling type daemon
by Corion (Patriarch) on Sep 13, 2017 at 11:28 UTC

    In addition to cron, see Schedule::Cron, which implements a cron daemon in Perl.

[reply]
[d/l]
[select]
Re: Scheduling type daemon
by Anonymous Monk on Sep 13, 2017 at 14:16 UTC
    On Linux, the systemd daemon has a very good and flexible built-in scheduler. It leaves cron in the dust.
[reply]
      Thought about that, but see above... really, the problem is the same as with cron. Looking for more control and granularity.
[reply]
Re: Scheduling type daemon
by Anonymous Monk on Sep 16, 2017 at 03:54 UTC
    Your schema should store:
    • last run time epoch milliseconds
    • frequency to run at, in seconds
    • other stuff like ids, foreign references, job names, etc.

    On startup, partition all your jobs by frequency (60 might mean something run every 60s; 3600 would be hourly; etc) and then sort by last run time, oldest to newest. Map each set to figure out each job's next run time.

    Look at the 0th item in each set to see which has the next run time. Sleep between now and then.

    When you wake up, execute your top job. Update its last run time. Add it back to the queue of jobs. Re-partition/sort/map the entire queue; calculate your next run time; rinse and repeat.

    There's plenty of opportunity here to improve this, by adding state tracking; offloading the sorting/mapping to your database; periodically refreshing the list of available jobs; etc. but for a naive implementation, this works.

[reply]

Back to Seekers of Perl Wisdom