There are packages that do job scheduling, including handling dependencies between jobs and recovery from failures. You might investigate some of them.
A fundamental issue is whether you want the various jobs to be aware of each other, or whether you want the dependencies and coordination to be handled externally, as a job scheduler does.
I would try to avoid polling. It is inefficient and/or introduces latency. If you don't want the package that builds the file to initiate the subsequent jobs when it is done, you might consider having the package that starts that package do so synchronously and run the following jobs when the first one exits.
To determine whether some other process has the file open, I would try to open the file and set an exclusive lock on it. If any other process has it open, the attempt to obtain an exclusive lock will fail. If the other process has an exclusive lock, you might not even be able to open the file.
You say "other jobs". Depending on how they interact, you may have to be careful to avoid deadlock, where every job is waiting on some resource held by some other job and nothing can progress. If each job uses (locks for exclusive access) only one resource at a time, this won't be a problem.
|