# orca.scheduler > A module of morca's API. The scheduler: fires schedules, senses datasets, queues backfills and rebuilds, and starts runs. ``` orca register ./build/taxi runs `taxi describe`, records the binary orca backfill taxi 2011-01..2024-12 a run per partition that isn't done orca backfill labels @all keyed: runs of keys_per_run undone keys orca rebuild taxi.trips 2024-01 the pipelines downstream, in order orca tick one pass, then exits orca scheduler a pass every second ``` The scheduler links no pipeline. It knows one by the `describe` it printed, kept in `orca.pipelines` with the binary's path, and reads the schedule, partitions and limits from that JSON in SQL. Everything it decides is in Postgres, so it can be stopped and restarted, or run twice, at any time: - **A fire is a row.** `orca.fires` is keyed by (pipeline, fire time), and a fire's row and its `queued` run are inserted in one transaction, so a fire is never made twice. Fires are counted from the last one recorded (or from when the pipeline was registered), so the fires missed while the scheduler was down are made once each, or, with `catchup=False`, only the latest. A fire whose partition is before the pipeline's first is recorded without a run. - **One live run per partition.** Every way a run is queued checks, under a lock on the pipeline's row, for a live run (`orca.live`) of the same keys: a fire whose partition a live run has records that run as its own, a sensor leaves such keys to it (the snapshot's run, if it left nothing else), a rebuild's level waits for it, the UI refuses to start one, and a backfill skips the key. A partition that's done is run again by a fire, sensor, rebuild or start, and its steps append again; `orca register` warns of a schedule that fires more than once for one partition. - **Starting a run is `resume`.** A queued run has nothing done, or what a resume or rerun from `orca ui` left it, which is what ` resume ` takes (with `--allow-new-code` when the resume said so). The scheduler marks the run `running`, with no heartbeat and itself as owner, and execs the binary, which claims it. It doesn't wait: the run records its own progress. - **`max_runs`.** Under a lock on the pipeline's row, the runs that are live (a fresh heartbeat, or started by the scheduler less than the stale limit ago) are counted, and only that many fewer than `max_runs` are started. - **Memory pressure** (`orca.pressure`, `orca.admission`). On the local runner and for control-placed runs, none starts while this host's PSI `some avg10` is over `ORCA_ADMIT_MEMORY_PRESSURE` (10%), each queued run `held` saying so; then the grants of a pipeline with a request (`orca.resources`). - **Backfills queue what isn't done.** A key is done when every mapped step has succeeded for it in a run of the pipeline's current code (`DONE_SQL`). A backfill drops the keys that are done or in a live run (queued, or running with its process alive: `live_sql`), sorts the rest and queues a run per `keys_per_run` of them; `orca backfill` says on stderr which keys it dropped and why. - **Where runs run.** `ORCA_RUNNER` is `local` (a process here, the default), `kubernetes` (a dista cluster of pods per run, `orca.kube`) or `ec2` (a head instance and worker instances per run, `orca.ec2`). A pipeline registered with `--placement control` runs here whatever the runner, within the control instance's limits (`orca.control`); one with `--placement ec2` waits on a scheduler whose runner isn't `ec2`. One with `--placement worker:` is never started here: a remote worker of the pool claims its queued runs over the UI's API (`orca.workers`), and the scheduler queues its due retries and its dead runs again for one, its fill runs too (held for a worker, `fill_max_runs` across the pool), and a run whose worker is gone whatever its trigger. - **Overlapping instances** (`orca.instances`, docs/overlap.md). With `ORCA_OVERLAP=on` a deploy starts the new image's scheduler beside the old one: the newest live instance leads and alone decides, and an older one starts nothing more, supervises its runs until they end (or kills them after `ORCA_DRAIN_SECONDS`) and exits. A local or control-placed run records the instance that started it; while that instance lists it the run is live (`orca.live.held_sql`), counted against `max_runs`, the control slots and the grants, and left alone. Once the instance is gone and the run's heartbeat stale, it's queued again whatever its trigger, audited. - **Sensors and rebuilds** (`orca.sensors`). A sensor pipeline's pass looks for snapshots of the datasets it reads, in `orca.commits` on every tick and from ` sense` every `ORCA_SENSE_SECONDS` (asked in a child the next ticks check on, killed after `ORCA_BINARY_SECONDS`), and queues a run, trigger `sensor`, for those it hasn't seen; with `keys_per_run` above 1 the keys go first to its queued sensor runs that haven't started, up to that, then to new runs of that many (#240). A keyed sensor's keys dataset is sensed too: a new snapshot of it queues the keys that aren't done, in runs of `keys_per_run`. Each rebuild queues its next level once the one before has succeeded. - **A snapshot it can't run** is seen all the same. One whose keys aren't partitions of the sensor's kind (a week in `orca.partitions`, or names for a dated sensor) is recorded in `orca.sensed` with no run and why, and the others the tick saw are run. A pass that fails, such as a `sense` that exits non-zero, is kept as the pipeline's `sense_error`, which `orca ui` shows, and printed once rather than on every tick. - **Fills** (`orca.fill`). A pipeline registered with `fills = "all"` in orca.toml has a second pass after its other runs': runs with trigger `fill`, `keys_per_run` partitions each, over the partitions of its range no run has taken (`fill_status`), up to `fill_max_runs` beside `max_runs`, none while its other runs are queued, it's paused, outside `fill_window` or over `fill_budget_per_day`. - **Retries.** A scheduled, backfilled, sensed or fill run that failed is started again after the pipeline's `retry_seconds` (else `ORCA_RETRY_SECONDS`, default 60) doubling with each start, at most its `max_retry_seconds` if set, plus up to a tenth more as jitter, up to `run_retries` times. One whose process died (its heartbeat went stale) is started again at once, within the same budget. A manual, UI or rerun run is never retried by the scheduler: one whose process died ends `failed`, with the error `interrupted: its process stopped`, so it can be resumed and holds its keys no longer. A run whose step raised `defer` ends `deferred` with `retry_at` the time it asked for, and is started again then, whatever its trigger; those starts (`deferrals`) don't count against `run_retries`. A run asked to cancel whose process died ends `cancelled`. - **New code.** `orca register` of a new build moves the queued runs to it, and the failed ones waiting for a retry that have no step done. A retry of a run with steps done by other code would be refused by the binary's `resume`, so the scheduler fails it at once instead of starting it, with an error (`runs.error`) saying to resume it with `--allow-new-code`. A local run whose process exits before claiming its run fails with the exit code and the last line of its driver log as its error; one whose process crashes after (a segfault, `abort`) has the attempts it left running failed with the crash's first frames (`orca.crash`). The scheduler's clock is `orca.civil.now`, which `ORCA_NOW` fixes; fire times and retry times are on it. Heartbeats are on the database's clock. A run's output goes to `///driver.log` when `ORCA_LOGS` is set: written there by a local run, published there when the process is reaped if `ORCA_LOGS` is in object storage, or kept from the driver pod's log on Kubernetes (`orca.kube`). Beside a local run's process the scheduler starts ` sample` (`orca.sample`), which writes the machine's metrics to `...//metrics/local.jsonl` until the process ends; `ORCA_SAMPLE_SECONDS=0` turns it off, on every runner. Apart from that, each tick reads each local and control-placed run's process tree in /proc, and once the process is reaped its attempts get their peak memory and CPU (`orca.usage`). `orca scheduler` gives each local and control-placed run a cgroup of its own where it can (`orca.cgroup`): a memory limit at its grant, an out-of-memory kill that fails its attempts saying so, and a retry that asks for more. ## Structs ### `Scheduler` ```mojo struct Scheduler ``` Implements: `Movable` Fields: - `db: State` - `stale: Float64` - `retry_seconds: Int64` - `sense_seconds: Float64` - `start_grace: Float64`: How long a run the scheduler started may go without a heartbeat. - `budget: Int`: `ORCA_VCPU_BUDGET` in thousandths of a vCPU; 0 for none. - `runner: Optional[KubeRunner]`: Set with `ORCA_RUNNER=kubernetes`. - `ec2: Optional[Ec2Runner]`: Set with `ORCA_RUNNER=ec2`; with neither, runs are local processes. - `pools: PoolScaler`: The pool autoscaler's state between passes (`orca.pool_scale`). - `logs: String` - `wait_for_binaries: Bool`: Whether a tick waits for the binaries it asks (`orca tick`, which has no later tick to collect them), rather than checking next tick. - `expire_seconds: Float64`: How often every dataset is expired by its own policy (`ORCA_EXPIRE_HOURS`, 24; 0 turns it off). - `maintain: String`: The `orca_maintain` binary that expires, or empty for none. - `fleet_seconds: Float64`: How often the fleet page's samples are read from the instances' files (`ORCA_FLEET_SECONDS`, 30; 0 turns it off). - `watchdog_seconds: Float64`: How often the watchdog's rules are checked (`ORCA_WATCHDOG_SECONDS`, 60; 0 turns it off): `orca.watchdog`. - `limits: ControlLimits`: What a control-placed run may use (`orca.control`). - `room: Room`: What this host can grant the runs of pipelines with a request (`orca.resources`): its capacity less its reserve. - `psi: Source`: Where this host's memory pressure is read from (`orca.pressure`), found once; none on macOS or without PSI. - `admission: Admission`: Whether runs may start on this host for its memory pressure, from each tick's reading. - `cgroups: Cgroups`: Where its runs' cgroups go (`orca.cgroup`); off until `contain`. - `instance: Int`: This scheduler's row of `orca.schedulers` (`orca.instances`), or 0 when it isn't an instance: `ORCA_OVERLAP` off, or `orca tick`. - `leading: Bool`: Whether this instance holds the lead (always, for no instance). - `draining: Bool`: Whether this instance drains: it starts nothing, and exits once the processes it started have ended. - `drain_seconds: Float64`: `ORCA_DRAIN_SECONDS` (3600): how long a drain waits for its runs before it kills what's left. - `finished: Bool`: Whether a drained instance has ended: `orca scheduler` exits. #### `__init__` ```mojo def __init__(out self, url: String) ``` #### `join` ```mojo def join(mut self) ``` With `ORCA_OVERLAP=on`, makes this scheduler an instance (`orca.instances`): it leads only once it holds the lead. #### `reconnect` ```mojo def reconnect(mut self, url: String) ``` A new connection after a tick failed, keeping what this instance supervises; the lead, which was the old connection's, is taken again on the next tick if no newer instance has it. #### `say_pressure` ```mojo def say_pressure(self) ``` Says once, for `orca scheduler`, where this host's memory pressure is read from and what it holds (`orca.pressure`). #### `contain` ```mojo def contain(mut self) ``` Turns on a cgroup per run (`orca.cgroup`) where this host lets orca make them, saying once which, and picks up the empty cgroups an earlier scheduler left. `orca scheduler` does this; `orca tick` and the other commands don't, as they don't reap what they start. #### `backfill` ```mojo def backfill(mut self, pipeline: String, keys: String, rerun: Bool = False, allow_append: Bool = False) -> Queued ``` Queues a run for each partition in `keys` that has none; see `backfill`. ## Aliases ### `AUTO_ONE_AT_A_TIME` ```mojo comptime AUTO_ONE_AT_A_TIME = "waiting: max_runs = \22auto\22 starts one run at a time until its runs have a request to weigh (cpus, memory_gb, learned or declared)" ``` Why a queued run of a `max_runs = "auto"` pipeline with no request waits while another runs. ### `RETRIED_SQL` ```mojo comptime RETRIED_SQL = "trigger IN ('schedule', 'backfill', 'sensor', 'fill')" ``` `_RETRIED`, for `orca.actions`.