Hey Paul,
A couple thoughts on how queues might help here. Requires node.
First, a queue can be setup to only run one job at a time, but also you can schedule the next job when the current one is finished. Basically you would not use the scheduler, you use the queues alone. Tobias has posted some code that can kick off the process on server start. You can also use this to control the delay before next job run. For example if the job does nothing, maybe you delay longer until it finds something to do.
But also, I have found that breaking long running jobs into smaller jobs ( often using a different queue) helps keep things manageable and efficient.