My colleagues use eHive for this sort of problem. (Link was seen as spam - try http://www.ensembl.org/info/docs/eHive/index.html)

It needs a database (mysql or sqlite) and a job submission system (e.g. LSF or SGE). Documentation is a little light, but it can handle the sorts of workflows you describe.