Docs

Scheduling and running

The scheduler starts runs when they're due, retries the ones that fail, and starts each on this machine or on a cluster of its own. Every run is recorded in Postgres, so any of them can be resumed, and its tables say what made them.

The scheduler

orca scheduler links no pipeline. It knows one by the describe that orca register recorded, and starts its runs as <binary> resume <run>, so a run records its own progress exactly as a run started by hand does. It needs ORCA_DATABASE_URL; the command line lists its commands.

./build/orca register ./build/gharchive
./build/orca backfill gharchive 2025-01-01..2025-01-31
./build/orca scheduler

Each fire is a row with a unique (pipeline, time), inserted with its queued run in one transaction, so two schedulers ticking at once, or one killed mid-tick, never fire twice. Fires missed while it was down are made on its next tick, once each. A queued run starts only while fewer than max_runs of the pipeline's runs are live, counted under a lock on the pipeline's row, and none start while it's paused. A scheduled, backfilled or sensed run that fails is started again after ORCA_RETRY_SECONDS, doubled with each start, plus up to a tenth as jitter, up to run_retries times; one whose process died is started again as soon as its heartbeat is stale. A run you started by hand, or from the UI, isn't retried. The scheduler holds nothing in memory, so it can be stopped and restarted at any time. Registering a new build moves the pipeline's queued runs to the new code.

On Kubernetes and EC2

By default the scheduler starts a run as a process on its own machine, and the run forks its Dista workers there. ORCA_RUNNER starts each run on a cluster of its own instead, from the image the pipeline was registered with:

./build/orca register \
    --image 123456789012.dkr.ecr.us-east-2.amazonaws.com/orca:data \
    --head-type c7g.large --worker-type c7g.2xlarge \
    --workers 3 ./build/taxi
ORCA_RUNNER=ec2 ./build/orca scheduler

ORCA_RUNNER=ec2 starts a head instance, on demand, running the Dista head and the driver (<binary> resume <run>), and --workers worker instances pointed at it; --spot makes the workers Spot instances, where the account allows them, and --workers MIN..MAX starts MIN and adds or drains workers up to MAX as the head's demand changes. Each instance is tagged with its run, told to terminate when it shuts down, and recorded with its type, when it started and ended, and why it ended (it shut itself down, it was terminated, or a Spot interruption). ORCA_RUNNER=kubernetes creates a driver Job, a head Pod and Service and the worker Pods in ORCA_RUNS_NAMESPACE, all labelled with the run and owned by the Job. A pipeline without Dista tasks gets only its driver.

Both clean up in layers. A run's instances or pods end when it does; every tick the scheduler removes those of runs that aren't running any more and those older than ORCA_RUN_DEADLINE, so nothing a killed scheduler left behind keeps running. A run whose head goes is started again on new instances. Before making a cluster, the scheduler adds up the vCPUs of every live run's pods or instances, and a run that would take them past ORCA_VCPU_BUDGET waits.

orca aws up makes all of this on AWS as one stack: a control instance running the scheduler with ORCA_RUNNER=ec2, Postgres and the UI, and what run instances need (orca aws, local or AWS?).

Placement and remote workers

A pipeline's placement says which machine starts its runs: default, the scheduler's runner; ec2, instances of their own; control, child processes of the scheduler on its own host, for small runs that shouldn't wait for an instance, within memory, CPU and count limits; or worker:<pool>, a remote worker of that pool. Set it in orca.toml (placement = "control") or with orca register --placement (placement.md).

A remote worker, such as a machine with spare CPU, runs orca worker --pool <pool> --remote <url>. It claims its pool's runs from orca ui over HTTPS and runs them on its own machine; each run records its state through the UI's API instead of Postgres, so nothing on the worker holds database credentials. Its token, from orca worker-token create --pool <pool>, is scoped to the pool and revocable. Since 0.15.0 a run whose worker is gone is queued again for the pool, whatever started it, within run_retries starts (Remote workers, workers.md).

./build/orca register --placement worker:bgent ./build/toy
./build/orca worker-token create --pool bgent        # on the server
ORCA_WORKER_TOKEN=orca_... ./build/orca worker --pool bgent --remote https://orca.example.com

Fills

With fills = "all" in orca.toml, the scheduler fills a pipeline's whole range by itself, one partition a run, in fill_order, with at most fill_max_runs at once, within an estimated fill_budget_per_day and in the UTC hours of fill_window. It never reruns a done partition and stops while the pipeline is paused. The pipeline page's Progress card shows done of total, rate, ETA and why a fill is idle (orca-toml.md#fills). Since 0.15.0 a pipeline placed on a worker pool fills too: its fill runs are queued in the same order and window for the pool's workers to claim, with at most fill_max_runs across the whole pool. From 0.24.0, fill_max_runs = "auto" starts as many as fit, as max_runs = "auto" does (orca.toml).

Memory and CPU

The scheduler measures every run it starts on its own host, and orca worker every run it starts: each attempt's peak memory, CPU time and peak cores, shown on the run, its steps and the pipeline's Resources card (since 0.14.0). Runs on EC2 and Kubernetes aren't measured this way; their machines are, under Machines. A pipeline can then ask for what it needs, in orca.toml or with orca register --cpus and --memory-gb, as the least it needs to start and the most it can use:

[pipelines.gharchive]
cpus = "1..16"
memory_gb = "2..24"
env = { DISTA_WORKER_SLOTS = "auto", GHARCHIVE_PARALLEL = "auto" }

Grants (since 0.14.0). On the local runner, for control-placed runs and on a remote worker, a run starts only if its minimum fits what the host can grant (its CPUs and memory less a reserve, ORCA_RESERVE_CPUS and ORCA_RESERVE_MEMORY_MB) less what the runs going there were granted; otherwise it waits, saying why (waiting: needs 2 GB, 1.3 GB free of grants). A run that starts gets up to its maximum of what's left, fixed for that start, in ORCA_GRANT_CPUS and ORCA_GRANT_MEMORY_MB. An env value "auto" lets the run size itself from the grant (orca.auto_count; DISTA_WORKER_SLOTS becomes the granted CPUs over the run's local workers). A pipeline without a request runs as before, and on EC2 and Kubernetes the request is kept but not used yet.

Memory pressure (since 0.15.0). Requests are estimates, so on the local runner, the control host and each worker no new run starts while Linux's pressure stall information reads some avg10 over ORCA_ADMIT_MEMORY_PRESSURE (10%); runs already going go on. A waiting run says waiting: memory pressure 23% (limit 10%). The hold ends under half the limit, and one that outlasts ORCA_ADMIT_PRESSURE_GRACE_SECONDS with none of morca's runs going raises an alert and lets runs start one at a time. Without PSI (macOS) nothing is held. Ops › Control shows the reading, and every run gets ORCA_MEMORY_PRESSURE_FILE to lower its own parallelism while it's high.

A cgroup per run (since 0.15.0). Where morca can make cgroup v2 cgroups, each local, control-placed and worker run gets one, its memory limit at its grant (else its request's maximum, else ORCA_RUN_MEMORY_MB), and the kernel kills it, it alone, ORCA_RUN_MEMORY_MARGIN (25) percent above that. The attempt fails out of memory at <limit>, and its retry and the pipeline's later runs ask for 1.5 times as much, up to memory_gb's maximum. Where cgroups aren't available (macOS, cgroup v1, no delegation, ORCA_CGROUPS=0) the scheduler says why once and runs as before; on the Hetzner server, orca-stack configure CGROUPS=1 turns them on (hetzner.md). orca-toml.md has all of it, and the roadmap what comes next.

The watchdog

Every ORCA_WATCHDOG_SECONDS the scheduler checks instance launches, estimated cost, idle workers, failing attempts, stuck runs and the control host's memory and disk. A rule past one of its thresholds pauses its pipelines, shows a red banner and sends an alert; it never resumes anything. Unpausing the pipelines clears the trip, orca watchdog ack keeps them paused, and orca watchdog snooze <rule> 2h silences a rule. On the local runner the cost rule reads "fixed price" and never trips (watchdog.md).

./build/orca watchdog status
control   memory 98.30 % (trips over 90.00, past), disk 10.60 % (trips over 85.00)  TRIPPED on memory

Alerts

With ORCA_NOTIFY_URL set to an ntfy topic, the scheduler pushes watchdog trips, runs that failed after their last retry, stalled fills, a full control disk and lost workers to your phone, each at most once per half hour, with a link into the UI. orca notify test checks it (alerts.md). For one server at a fixed price, the Hetzner runbook sets all of this up.

Runs in Postgres

With ORCA_DATABASE_URL set, the binary creates and migrates an orca schema at start-up and records every run, every step of every partition, and every attempt, with the inputs and outputs of each. Each change of state is one transaction.

export ORCA_DATABASE_URL=postgresql:///orca
./build/gharchive run --partitions 2025-01-01..2025-01-03
./build/gharchive status 7        # run 7's steps and attempts
./build/gharchive resume 7

resume reruns every step that hasn't succeeded, in order, with its recorded input; steps that succeeded are read back, not run again. A run started by a binary with a different registry hash is refused unless --allow-new-code is passed.

One process owns a run at a time. It holds the run with a heartbeat, written from a thread with its own connection, and resume refuses a run whose heartbeat is younger than ORCA_STALE_SECONDS. A run whose process was killed goes stale and can be resumed; a run that ended can be resumed at once. Each attempt also checks that its process still owns the run, so a process that lost its run stops at its next step rather than racing the new owner. tests/kill.sh kills a run at a random step with kill -9, resumes it, and checks every step ran to success exactly once.

Postgres must keep synchronous_commit on: with it off, an immediate restart of the server can lose commits it already acknowledged. The chaos test from PR #6 found this when a run's row vanished.

Deploys and rollbacks

On the Hetzner stack, deploy.sh <tag> replaces only the containers that changed, the UI and apps without downtime, and pauses the pipelines only when it replaces the scheduler, whose restart would kill its runs. It backs the database up first, and deploy.sh --rollback goes back to the tags deployed before; an older orca runs on a newer schema whose migrations since are additive. Deploying has the rest: overlapping schedulers, settings and SOPS secrets, and rescaling.

Logs

With ORCA_LOGS set to a directory, each attempt writes <pipeline>/<run>/<step>/<partition>/<attempt>.log: its start, its end, its error, and its ctx.log lines with UTC timestamps. The path is recorded with the attempt and shown by status <run>. A run with Dista tasks writes the runtime's timeline beside them as timeline.json, in Chrome's trace format, which Perfetto opens. The scheduler keeps a run's driver output as driver.log, and an EC2 run's head and workers upload theirs to cluster/<role>-<instance>.log every ORCA_CLUSTER_LOG_SECONDS while they run, as an instance can go without notice.

ORCA_LOGS and ORCA_WAREHOUSE can be locations in object storage, s3://bucket/prefix, configured from the AWS_* environment or the instance's role. An object can't be appended to, so a log is written to a file under ORCA_LOG_STAGE and uploaded when it's complete: an attempt's when the attempt ends, a timeline when the run does. The UI reads them back from wherever they are.

Machine metrics

Beside every process doing a run's work, the runner starts <binary> sample, which writes a JSON line of the machine's CPU, memory, disk, disk I/O and network every ORCA_SAMPLE_SECONDS to <pipeline>/<run>/metrics/<instance>.jsonl under ORCA_LOGS; since 0.15.0 orca worker does the same beside each run it starts, as metrics/worker-<name>.jsonl. A run's page draws them as its Machines section, on the same time axis as its steps, and marks the stretches where memory use or a disk was over 90% or CPU steal over 20%, which explain most slow or failed runs. They are the machine's numbers; the run's own peak memory and CPU are in its Resources card.

Lineage

Every read and commit a step makes is recorded with its dataset, so lineage is two walks over them. Back from a snapshot: the run, step, partition and code that made it, the snapshots that attempt read, what made each of those, and so on, to sources no morca run made. Forward: everything made from it. Both stop after 20 steps.

$ ./build/orca lineage lake.report
lake.report@3141592653
made from:
  0 lake.report@3141592653 by report run 9 report code 0x5e1f... from lake.clean@2718281828
  1 lake.clean@2718281828 by clean run 8 clean 2024-02 code 0x9a0c... from lake.raw@1618033988
  2 lake.raw@1618033988 by ingest run 6 load 2024-02 code 0x77d2...
made from it:
  nothing

Rebuilds

$ ./build/orca rebuild raw.events 2024-03
rebuild 1
  level 0 clean 2024-03
  level 1 report 2024-03

A rebuild runs every pipeline downstream of a dataset, a sensor or not, over the partitions given. Each pipeline is placed at its longest distance from the dataset, so one that reads two rebuilt datasets runs after both, and the scheduler queues a level once every run before it has succeeded. A run that fails stops the rebuild; resuming it from the UI carries on. The rebuild's own commits don't also start the sensors it reruns, so each pipeline runs once.

Environment

When things fail

A step that raises is retried up to its retries, then fails. The steps downstream of it are skipped, while other partitions' steps that don't depend on it carry on, so one bad month doesn't stop the others; the run ends failed and can be resumed. A process that dies leaves its run running with a stale heartbeat; resume marks the interrupted attempt failed and continues from there. One that crashed has its attempts failed with its stack dump's first frames as soon as its runner sees it end (Debugging failures). A run over its memory, in a cgroup of its own, is killed alone and retried with 1.5 times the memory (Memory and CPU). A commit that landed before the crash is found by the retry rather than made again. Fuzzing, and a chaos test that kills runs, restarts Postgres and races two resumes over many seeds, run in CI.