orca.scheduler

orca register ./build/taxi             runs `taxi describe`, records the binary
orca backfill taxi 2011-01..2024-12    a run per partition that isn't done
orca backfill labels @all              keyed: runs of keys_per_run undone keys
orca rebuild taxi.trips 2024-01        the pipelines downstream, in order
orca tick                              one pass, then exits
orca scheduler                         a pass every second

The scheduler links no pipeline. It knows one by the describe it printed, kept in orca.pipelines with the binary's path, and reads the schedule, partitions and limits from that JSON in SQL. Everything it decides is in Postgres, so it can be stopped and restarted, or run twice, at any time:

• A fire is a row. orca.fires is keyed by (pipeline, fire time), and a fire's row and its queued run are inserted in one transaction, so a fire is never made twice. Fires are counted from the last one recorded (or from when the pipeline was registered), so the fires missed while the scheduler was down are made once each, or, with catchup=False, only the latest. A fire whose partition is before the pipeline's first is recorded without a run.
• One live run per partition. Every way a run is queued checks, under a lock on the pipeline's row, for a live run (orca.live) of the same keys: a fire whose partition a live run has records that run as its own, a sensor leaves such keys to it (the snapshot's run, if it left nothing else), a rebuild's level waits for it, the UI refuses to start one, and a backfill skips the key. A partition that's done is run again by a fire, sensor, rebuild or start, and its steps append again; orca register warns of a schedule that fires more than once for one partition.
• Starting a run is resume. A queued run has nothing done, or what a resume or rerun from orca ui left it, which is what <binary> resume <run> takes (with --allow-new-code when the resume said so). The scheduler marks the run running, with no heartbeat and itself as owner, and execs the binary, which claims it. It doesn't wait: the run records its own progress.
• max_runs. Under a lock on the pipeline's row, the runs that are live (a fresh heartbeat, or started by the scheduler less than the stale limit ago) are counted, and only that many fewer than max_runs are started.
• Memory pressure (orca.pressure, orca.admission). On the local runner and for control-placed runs, none starts while this host's PSI some avg10 is over ORCA_ADMIT_MEMORY_PRESSURE (10%), each queued run held saying so; then the grants of a pipeline with a request (orca.resources).
• Backfills queue what isn't done. A key is done when every mapped step has succeeded for it in a run of the pipeline's current code (DONE_SQL). A backfill drops the keys that are done or in a live run (queued, or running with its process alive: live_sql), sorts the rest and queues a run per keys_per_run of them; orca backfill says on stderr which keys it dropped and why.
• Where runs run. ORCA_RUNNER is local (a process here, the default), kubernetes (a dista cluster of pods per run, orca.kube) or ec2 (a head instance and worker instances per run, orca.ec2). A pipeline registered with --placement control runs here whatever the runner, within the control instance's limits (orca.control); one with --placement ec2 waits on a scheduler whose runner isn't ec2. One with --placement worker:<pool> is never started here: a remote worker of the pool claims its queued runs over the UI's API (orca.workers), and the scheduler queues its due retries and its dead runs again for one, its fill runs too (held for a worker, fill_max_runs across the pool), and a run whose worker is gone whatever its trigger.
• Overlapping instances (orca.instances, docs/overlap.md). With ORCA_OVERLAP=on a deploy starts the new image's scheduler beside the old one: the newest live instance leads and alone decides, and an older one starts nothing more, supervises its runs until they end (or kills them after ORCA_DRAIN_SECONDS) and exits. A local or control-placed run records the instance that started it; while that instance lists it the run is live (orca.live.held_sql), counted against max_runs, the control slots and the grants, and left alone. Once the instance is gone and the run's heartbeat stale, it's queued again whatever its trigger, audited.
• Sensors and rebuilds (orca.sensors). A sensor pipeline's pass looks for snapshots of the datasets it reads, in orca.commits on every tick and from <binary> sense every ORCA_SENSE_SECONDS (asked in a child the next ticks check on, killed after ORCA_BINARY_SECONDS), and queues a run, trigger sensor, for those it hasn't seen; with keys_per_run above 1 the keys go first to its queued sensor runs that haven't started, up to that, then to new runs of that many (#240). A keyed sensor's keys dataset is sensed too: a new snapshot of it queues the keys that aren't done, in runs of keys_per_run. Each rebuild queues its next level once the one before has succeeded.
• A snapshot it can't run is seen all the same. One whose keys aren't partitions of the sensor's kind (a week in orca.partitions, or names for a dated sensor) is recorded in orca.sensed with no run and why, and the others the tick saw are run. A pass that fails, such as a sense that exits non-zero, is kept as the pipeline's sense_error, which orca ui shows, and printed once rather than on every tick.
• Fills (orca.fill). A pipeline registered with fills = "all" in orca.toml has a second pass after its other runs': runs with trigger fill, keys_per_run partitions each, over the partitions of its range no run has taken (fill_status), up to fill_max_runs beside max_runs, none while its other runs are queued, it's paused, outside fill_window or over fill_budget_per_day.
• Retries. A scheduled, backfilled, sensed or fill run that failed is started again after the pipeline's retry_seconds (else ORCA_RETRY_SECONDS, default
  60. doubling with each start, at most its max_retry_seconds if set, plus up to a tenth more as jitter, up to run_retries times. One whose process died (its heartbeat went stale) is started again at once, within the same budget. A manual, UI or rerun run is never retried by the scheduler: one whose process died ends failed, with the error interrupted: its process stopped, so it can be resumed and holds its keys no longer. A run whose step raised defer ends deferred with retry_at the time it asked for, and is started again then, whatever its trigger; those starts (deferrals) don't count against run_retries. A run asked to cancel whose process died ends cancelled.
• New code. orca register of a new build moves the queued runs to it, and the failed ones waiting for a retry that have no step done. A retry of a run with steps done by other code would be refused by the binary's resume, so the scheduler fails it at once instead of starting it, with an error (runs.error) saying to resume it with --allow-new-code. A local run whose process exits before claiming its run fails with the exit code and the last line of its driver log as its error; one whose process crashes after (a segfault, abort) has the attempts it left running failed with the crash's first frames (orca.crash).

The scheduler's clock is orca.civil.now, which ORCA_NOW fixes; fire times and retry times are on it. Heartbeats are on the database's clock. A run's output goes to <ORCA_LOGS>/<pipeline>/<run>/driver.log when ORCA_LOGS is set: written there by a local run, published there when the process is reaped if ORCA_LOGS is in object storage, or kept from the driver pod's log on Kubernetes (orca.kube). Beside a local run's process the scheduler starts <binary> sample (orca.sample), which writes the machine's metrics to .../<run>/metrics/local.jsonl until the process ends; ORCA_SAMPLE_SECONDS=0 turns it off, on every runner. Apart from that, each tick reads each local and control-placed run's process tree in /proc, and once the process is reaped its attempts get their peak memory and CPU (orca.usage). orca scheduler gives each local and control-placed run a cgroup of its own where it can (orca.cgroup): a memory limit at its grant, an out-of-memory kill that fails its attempts saying so, and a retry that asks for more.