Reference

orca.toml

orca.toml sits at the root of a project's repository and says how each pipeline runs: where, on what, how much it uses and how it fills. The pipeline's code still says what it does. orca register reads the file, checks every key, and records each value with where it came from.

An example

[defaults]
placement = "ec2"
head_type = "t4g.medium"

[pipelines.gharchive]
worker_type = "c7g.xlarge"
workers = "0..4"
max_runs = 2
env = { DISTA_WORKER_SLOTS = "4", GHARCHIVE_PARALLEL = "16" }
fills = "all"                # fill the whole range by itself
fill_max_runs = 2
fill_budget_per_day = 5.0    # estimated US dollars a UTC day
fill_window = "22..6"        # UTC hours fill runs start in
fill_lag = "2h"              # a day fills 2 hours after it ends

[pipelines.mojo_repos]
placement = "worker:lab"
secrets = { GITHUB_TOKEN = "orca/github-token" }
cpus = "1..2"                # starts once 1 CPU and 0.5 GB fit,
memory_gb = "0.5..2"         # and is given up to 2 of each

[pools.lab]
provider = "hetzner"
machines = [{ type = "cax31", wait = 600 }, { type = "cpx32" }]
locations = ["hel1", "nbg1"]
max_machines = 3

The full reference, with every rule, is docs/orca-toml.md; the code that reads the file is config.mojo and pool_config.mojo.

Tables

Any other table, and any key at the top level, is refused. morca reads the part of TOML 1.0 a config needs: comments, dotted and quoted keys, basic and literal strings, decimal integers, floats, booleans, arrays (over several lines, with a trailing comma), inline tables and arrays of them. Multi-line strings, dates, [[arrays of tables]], hexadecimal, octal and binary integers, inf and nan are refused with a message that says so, as is a key or table defined twice.

[pipelines.<name>]

Every key below can go in [defaults] or in a pipeline's own table. A value comes from the first of these that sets it: a flag to orca register (recorded as an override), the pipeline's table, [defaults], the environment, then the code. env and secrets merge one name at a time. image and --spot stay flags only, and no flag sets the fill* keys or learn.

Placement and size

placement.md describes the four placements, and Remote workers covers worker:<pool>.

Requests

Fills

Secrets and environment

How many at once

max_runs = "auto" starts another run of the pipeline whenever its request fits in what this host's running runs weren't granted, up to a ceiling of 64; "auto..N" sets the ceiling to N. A bigger host then runs more at once with no change to the file: at 2 GB a run, 4 on a host with 8 GB to grant, 12 on one with 24.

[pipelines.gharchive]
memory_gb = "auto"       # learned from its runs
max_runs = "auto"
fill_max_runs = "auto"   # from 0.24.0

It needs a request to weigh, declared or learned. Until it has one, it starts one run at a time, and its queued runs say so. On EC2, Kubernetes and remote workers, where there is no host's room to weigh, the ceiling is the limit; a worker still admits what it takes against its own host. From 0.24.0, fill_max_runs = "auto" does the same for fill runs (migration 55 records it as orca.pipelines.fill_max_runs_auto). An older morca reads "auto" as its ceiling.

Requests and grants

cpus and memory_gb are a request: the least a run needs to start and the most it can use (a single number is both). On the local runner, for control-placed runs and on remote workers, a run starts only if its minimum fits what the host can grant, less what the runs going there were granted. The host's grant is its CPUs and memory (its cgroup's limits if set, or ORCA_HOST_CPUS and ORCA_HOST_MEMORY_MB) less ORCA_RESERVE_CPUS (1) and ORCA_RESERVE_MEMORY_MB (1536); a worker's reserve is 0 and 512. A run that starts gets up to its maximum of what's left, fixed for that start, in ORCA_GRANT_CPUS and ORCA_GRANT_MEMORY_MB.

• A pipeline without a request runs as before, unchecked and not counted against the others, so on a shared host give every pipeline one.
• On EC2 and Kubernetes the request is kept but not used yet.
• Where morca can make cgroups, a run's grant is its memory limit, and the kernel kills it ORCA_RUN_MEMORY_MARGIN (25) percent above that. Its retry and later runs then ask for ORCA_RUN_MEMORY_GROWTH (1.5) times as much, up to memory_gb's maximum.
• "auto" takes the request from the pipeline's last 20 successful measured runs: memory at their 95th percentile plus 15%, CPUs at the 90th percentile of peak cores. With fewer than 3 such runs there is no request yet. "auto..24" caps it at 24.
• An env value "auto" lets the run size itself from its grant (orca.auto_count). DISTA_WORKER_SLOTS = "auto" becomes the granted CPUs over the run's local Dista workers, or 0 (one per core) without a grant.

Scheduling and running explains this in prose, and orca-toml.md has every rule.

Memory pressure

On top of max_runs, fill_max_runs and grants, no new run starts on the local runner, the control host or a worker while Linux's pressure stall information reads some avg10 over ORCA_ADMIT_MEMORY_PRESSURE (10%; 0 turns it off). Runs already going go on. The hold ends under half the limit, or after 30 s at most at the limit. One that lasts ORCA_ADMIT_PRESSURE_GRACE_SECONDS (600) with none of morca's runs going raises the memory-pressure alert and lets runs start one at a time. Without PSI, as on macOS, nothing is held. These are environment variables, not orca.toml keys.

env and secrets

• A pipeline's env goes only to its own runs: their process environment on the local runner, the control host and a worker, each pod's env on Kubernetes, and each instance's script and containers on EC2.
• ORCA_RUN_ENV still copies the scheduler's named variables to every run; a pipeline's env wins for a name it sets.
• On EC2 the env is part of the warm pool's key, so changing it launches new instances.
• Stack settings (database, warehouse, logs, runner, budget) stay in the scheduler's environment file, not in env.
• secrets are read with ORCA_SECRET_FETCH when it is set (<command> <id> through sh -c), else from Secrets Manager. IDs always start with orca/.

Fills

With fills = "all", on every tick and after the pipeline's other runs have started, the scheduler queues fill runs over its range: its first partition through the last one that ended at least fill_lag ago. Each fill run takes keys_per_run partitions in fill_order, with trigger fill. A fill never runs a partition twice: one done in any of the pipeline's code, in a live run, or ever queued as a fill is taken. Fill runs don't count against max_runs, never start while other runs of the pipeline are queued, and on a worker pool are left for the pool's workers to claim, with fill_max_runs across the whole pool.

The cost estimate fails closed: an hour of the head plus the most workers at on-demand prices, times 1.2, against today's spend on fill runs' instances. Runs on the control host or a local process cost nothing and need no budget. Keyed pipelines, sensors and pipelines with no first partition can't fill, and orca register refuses fills = "all" for them. The pipeline page's Fill line and fill in its JSON say why a fill is idle:

[pools.<name>]

A [pools.<name>] table makes a pool of remote workers that orca grows and shrinks itself. Pipelines run on it with placement = "worker:<name>", as on workers started by hand. orca register records the file's pools in orca.pools; a pool the file no longer has is deleted there, and its machines go once idle.

[pools.lab]
provider = "hetzner"
machines = [
  { type = "cax31", wait = 600 },   # prefer this; keep trying it for 10 min
  { type = "cax21", wait = 600 },
  { type = "cpx32" },               # the last: no wait
]
locations = ["hel1", "nbg1", "fsn1"]
max_machines = 3
idle_seconds = 900

• Grow. While runs of the pool's pipelines have waited 20 s that a worker could start, no machine is still booting and the pool has fewer than max_machines, the leading scheduler makes one. It tries the current type in each location in turn for its wait, then the next type, and skips a type whose architecture the worker image lacks. From 0.24.0, a type the provider refuses for the project's quota (Hetzner's resource_limit_exceeded) moves it on to the next usable type at once.
• Boot. A machine holds only a worker token of its own and asks /api/worker/boot for the image, its login and the bucket's variables. From 0.24.0 machines are named orca-pool-<pool>-<n>.
• Shrink. A machine idle for idle_seconds with nothing waiting is removed, its token revoked first. So is one whose worker never shows up (15 min) or goes silent (5 min), and one with the pool's labels that morca doesn't know.
• A running machine is never replaced by a type higher in the list: the order only matters when morca adds one.

What the scheduler and the UI need in their environment for pools:

Pools and autoscaling covers running them.

[apps.<name>]

From 0.26.0, an app is a long-running binary of the project's image, /opt/morca/bin/<name>, that runs beside the UI: the census console is one. Its table says what the stack gives it, as a pipeline's table says how its runs run. Which apps a stack runs is the stack's choice (Apps on the deploy page).

[apps.console]
port = 8080                                   # $PORT
check = "/healthz"                            # 200 once it's ready
env = { CONSOLE_SEMANTIC = "1" }
lake = true                                   # the warehouse and the stack's S3 keys
secrets = { POSTHOG_KEY = "orca/posthog-key" } # an ID, never the value
memory_gb = 1

• Nothing it doesn't ask for. An app never gets morca's database, and gets the lake only with lake = true.
• Names. Lower-case letters, digits, _ and -, and not one of the stack's own containers (scheduler, ui, sampler, cloudflared, postgres).
• Overlays. A stack's overlay can change an app the file declares, but can't add one.

morca config apps [<file>] [--overlay F] prints the apps one fact per line, as the stack's script reads them, with secrets by ID:

$ morca config apps
console port 8080
console check /healthz
console lake true
console memory_mb 1024
console env CONSOLE_SEMANTIC=1
console secret POSTHOG_KEY orca/posthog-key

What is refused

Each refusal names the file and line, and nothing is registered:

orca.toml:7: [pipelines.gharchive] has no key workrs (it has placement, head_type, ...)
orca.toml:3: [pipelines.toy] workers: not a count or a range MIN..MAX: 4..2

• Unknown keys and tables, and a value of the wrong type.
• A workers, cpus or memory_gb range like "4..2"; cpus = 0, cpus = 1.5 or memory_gb = 0; auto anywhere but alone or as a range's lower end, or "auto.." without its maximum.
• A secret ID outside orca/.
• A fill_window like "6..6" or "1..30", a negative fill_budget_per_day, or fill_max_runs = 0.
• A fill_lag without a unit, with an unknown one ("2w"), or over 31d.
• fills = "all" for a keyed pipeline, a sensor, or one with no first partition.
• An env name that isn't a valid variable name, that looks like a secret (*TOKEN*, *SECRET*, *PASSWORD*, *DATABASE_URL*), or that morca sets itself: ORCA_DATABASE_URL, ORCA_IMAGE, ORCA_SECRETS, ORCA_WORKERS, DISTA_ROLE, DISTA_HEAD, DISTA_ADVERTISE, ORCA_GRANT_CPUS, ORCA_GRANT_MEMORY_MB, ORCA_MEMORY_PRESSURE_FILE.
• A pool without provider or machines, a provider other than hetzner or ec2, a machine entry with a key other than type and wait, a type that isn't 1 to 64 of a-z, 0-9, ., - and _, a type listed twice, a missing wait on any entry but the last, a Hetzner pool without locations, or max_machines or idle_seconds out of range.

How orca register reads it

• A pipeline the file names is registered with exactly what the file and flags say. A key removed from the file goes back to its default, and a flag applies to that registration only.
• A pipeline the file doesn't name keeps what was registered for a flag left out, and orca register warns that it isn't in the file.
• Without paths, orca register registers every [pipelines.<name>] as <dir>/<name>, where the directory is ORCA_PIPELINES_DIR or pipelines/ next to the file. A missing binary is an error, and nothing is registered. A project's image sets ORCA_CONFIG, ORCA_PIPELINES_DIR, ORCA_COMMIT and ORCA_SOURCE, so orca register alone registers the whole file.
• orca aws register takes the same --config and --no-config, checks the file locally, and sends it whole to the control instance with its git commit.
• What's recorded: orca.pipelines.config keeps the file's name, SHA-256, git commit and whether it had uncommitted changes, the flags that overrode it, and the source of every value.

orca config show, check and export

$ orca config show gharchive
gharchive  registered from orca.toml @ 3f2a9c1d0b7e, sha256 b0e8af31ea3e
  overrides: --workers 0..8
  placement               ec2         orca.toml [defaults]
  worker_type             c7g.xlarge  orca.toml [pipelines.gharchive]
  workers                 0..8        flag --workers
  keys_per_run            1           code default (the pipeline's keys_per_run)
  env.DISTA_WORKER_SLOTS  4           orca.toml [pipelines.gharchive]

• orca config show [pipeline] prints each registered value with its source: code default, orca.toml, flag or environment, and a learned request after the settings. --json prints one object, or {"pipelines": [...]} without a name. The pipeline page's Configuration card and config in /api/pipelines/<name> show the same.
• orca config check [file] [--secrets] refuses what orca register would, with no database, for a project's CI. It warns about an instance type missing from morca's price table, and with --secrets reads each secret without printing it. It exits 1 on any problem.
• orca config export [--out -] [--force] writes an orca.toml with a table per registered pipeline, holding what is registered beyond the defaults, so registering from it gives the same registration back. It writes no [defaults]. Use it to move from register flags to a file.