orca.pool

<binary> pool-wait <instance> <run> <start>             a worker: until its run is over
<binary> pool-next <instance> <run> <start> <seconds>   until it's handed another run

The scheduler hands an idle instance to a run by ending its row in orca.instances and inserting one for the new run, pooled, in one transaction. So the instance learns of its next run from its own rows, and Postgres, which every instance reaches already, is the only channel: no other service, port or permission. pool-next marks the row of the run the instance served idle (its work has stopped: the head's driver and dista head have exited, a worker's processes are killed), and beats polled every second, which the scheduler checks before it hands the instance out. When a new row appears, it records that it joined, prints next <run> <start> <head address> <0|1> (the last, whether the run may use new code) and exits 0; when its row ends with no other (the scheduler shut the idle instance down), or seconds and a minute more pass, it exits 1 and the script shuts the instance down.

pool-wait, beside a worker's processes, exits 0 once the run isn't running at that start any more, or the worker's row has ended: dista workers don't end with their head, so this is how a worker knows to stop. Both ride out a database that can't be reached for a while, retrying every second.