Looking for a POE based distributed job scheduler

jfroebe has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I have a virtual machine that runs several hundred jobs currently through cron. We're reaching the limit of what one vm can handle (nope, I don't have a choice of a physical machine). Response time during peak activity is affecting tasks so it is time to add another vm and load balance the work.

What I would like to do is: A Perl/POE based distributed job server that will run arbitrary jobs on a load based, round robin, or designated server manner. The job server must either be a Master/Slave (High Availability) or a peer2peer so if the job server(s) are unavailable, the jobs continue. Has anyone seen such a beast using Perl/POE?

I'm asking because I really don't want to reinvent the wheel if I don't have to.

Jason L. Froebe

Blog, Tech Blog

Comment on Looking for a POE based distributed job scheduler

Replies are listed 'Best First'.
Re: Looking for a POE based distributed job scheduler by merlyn (Sage) on Dec 29, 2009 at 02:03 UTC
Beanstalk::Client? -- Randal L. Schwartz, Perl hacker The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.	[reply]
Re^2: Looking for a POE based distributed job scheduler by jfroebe (Parson) on Dec 29, 2009 at 02:55 UTC
Possible. I was hoping a POE based solution for easier expansion/control. It'll be on the list for a proof of concept :) Jason L. Froebe Blog, Tech Blog	[reply]
Re^3: Looking for a POE based distributed job scheduler by suaveant (Parson) on Dec 29, 2009 at 14:19 UTC
Beanstalk seems to be very basic but elegant, but you could probably make it work by having intelligent clients that pulled jobs from the queue when your resource criteria were met. - Ant - Some of my best work - (1 2 3)	[reply]
Re: Looking for a POE based distributed job scheduler by suaveant (Parson) on Dec 28, 2009 at 20:48 UTC
I don't know of anything like that but a while ago I was playing with POE a bit on a resource based job scheduler that might at least get you started in the right direction. You are welcome to it if you want. It was pretty much a proof of concept so don't expect great great things, but it looked promising. - Ant - Some of my best work - (1 2 3)	[reply]
Re^2: Looking for a POE based distributed job scheduler by jfroebe (Parson) on Dec 28, 2009 at 21:44 UTC
thanks! :) Looks like that might help for part of it but... the more I think about this the more I want the p2p job scheduler. One idea that looks like it might have hope is: A message bus (possibly POE::Component::MessageQueue) that will allow job queuing, job statuses, etc to be communicated to the N nodes Static job list, statuses, etc via a replicated db. Possibly using Net::CouchDB - this would also be used as the backend to provide statuses to a front end of some sort (web or similar). Configuration of the peer job 'servers' via a simple ini file (Config::Simple) Thoughts on this hair brained idea? Jason L. Froebe Blog, Tech Blog	[reply]
Re^3: Looking for a POE based distributed job scheduler by suaveant (Parson) on Dec 29, 2009 at 12:35 UTC
My thoughts would be that you'd probably save yourself a lot of effort by setting up ssh keys on each system and doing all the work on the main site. That way you don't have to do nearly as much synchronization and maintaining of multiple running instances that all need to communicate. Just use remote ps calls, remote run your apps (which makes them local instances sort of) and check /proc/loadavg for system load info. Don't make things harder than you have to. Then the only actual hard part you need to write is the part that decides when to use which computer and how to decide your resource allocation. (my code also handles time based allocation i.e. don't run until at least 4pm) - Ant - Some of my best work - (1 2 3)	[reply]
Re: Looking for a POE based distributed job scheduler by stonecolddevin (Parson) on Dec 29, 2009 at 19:32 UTC
gearman is nice like this, and has the option for a POE backend. mtfnpy	[reply]
Re^2: Looking for a POE based distributed job scheduler by jfroebe (Parson) on Dec 30, 2009 at 16:06 UTC
At first glance it looks like gearman may just well fit Need to do a proof of concept on it to verify but thanks! :) Jason L. Froebe Blog, Tech Blog	[reply]