Operations

Sizing and scaling

morca scales in two directions. Vertically, each run is sized to its share of the host it runs on. Horizontally, more runs go at once, on more machines, and one run can spread over several. This page says how each works and which setting to change.

Vertical: sizing a run

Requests

cpus and memory_gb are a pipeline's request: the least it needs to start and the most it can use, written "MIN..MAX" (a single number is both).

[pipelines.gharchive]
cpus = "1..16"
memory_gb = "2..24"
env = { DISTA_WORKER_SLOTS = "auto", GHARCHIVE_PARALLEL = "auto" }

A pipeline without a request starts as before: no check and no grant. Its runs aren't counted against the others either, so on a shared host give every pipeline a request, or keep the host's reserve large enough for the ones without. On EC2 and Kubernetes, whose runs get machines of their own, the request is kept but not used yet.

Admission and grants

On the local runner, for control-placed runs and on each remote worker, a run starts only if its minimum fits what the host can grant, less what the runs already going there were granted. What a host can grant is its CPUs and memory (its cgroup's limits if set; ORCA_HOST_CPUS and ORCA_HOST_MEMORY_MB override them), less a reserve: ORCA_RESERVE_CPUS and ORCA_RESERVE_MEMORY_MB, 1 CPU and 1536 MB on the scheduler's host and 0 and 512 MB on a worker. A run that doesn't fit stays queued and says why: waiting: needs 2 GB, 1.3 GB free of grants.

A run that starts gets up to its maximum of what is left, in ORCA_GRANT_CPUS and ORCA_GRANT_MEMORY_MB. The grant is fixed for that start: nothing shrinks or is killed to make room, and a retry is granted again. An env value "auto" lets the pipeline size itself from its grant (orca.auto_count), and DISTA_WORKER_SLOTS = "auto" becomes the granted CPUs over the run's local dista workers.

Learned requests

From the peaks of a pipeline's last 20 successful measured runs, morca works out the request they suggest: memory is the 95th percentile of their peak memory plus 15%, rounded up to a quarter GB; CPUs are the 90th percentile of their peak cores, rounded up. With fewer than 3 such runs there is none. memory_gb = "auto" or cpus = "auto" uses it, and "auto..24" uses it with 24 as the most it may be granted.

[pipelines.gharchive]
memory_gb = "auto"     # learned from its runs
max_runs = "auto"

Until there is a recommendation, that resource has no request. The pipeline's Resources card and orca config show show the recommendation either way (learned from 20 runs: 3.25 GB, 2 CPUs), and learn = false keeps auto from applying it. Before rolling back to a morca before 0.21.0, register without auto: an older morca refuses it (Learned requests).

A cgroup per run

Where morca can make cgroup v2 cgroups, each local, control-placed and worker run gets one. Its grant is the cgroup's memory.high, and ORCA_RUN_MEMORY_MARGIN (25) percent above it the kernel kills the run, and only that run. A run held at its limit with its tasks stalled on memory for ORCA_RUN_MEMORY_HELD_SECONDS (120) is killed by morca the same way. The attempt fails out of memory at <limit>, and the retry and the pipeline's later runs ask for ORCA_RUN_MEMORY_GROWTH (1.5) times as much, up to memory_gb's maximum. Killed at that maximum already, the run fails without a retry. A run without a request is limited to ORCA_RUN_MEMORY_MB, by default what the host can grant. On the Hetzner server, orca-stack configure CGROUPS=1 turns cgroups on (hetzner.md).

Memory pressure

Requests are estimates, and a host can run short of memory while its grants still add up: page cache, Postgres, a run past its estimate. So on top of the grants, the scheduler reads its host's memory pressure every tick, and each worker reads its own every poll. This is Linux's pressure stall information (PSI): the share of time tasks were stalled waiting for memory.

• The hold. While some avg10 is over ORCA_ADMIT_MEMORY_PRESSURE (10 percent; 0 turns it off), no new run starts on that host. Runs already going go on. A queued run says waiting: memory pressure 23% (limit 10%), with on worker <name> on a pool. The hold ends once the reading is under half the limit, or has stayed at most the limit for 30 seconds.
• Never forever. A hold that lasts ORCA_ADMIT_PRESSURE_GRACE_SECONDS (600) with none of morca's runs going on the host isn't morca's doing. The scheduler or worker then raises the memory-pressure alert once and starts runs there one at a time until the pressure eases.
• Which file. ORCA_MEMORY_PRESSURE_FILE if set; else the memory.pressure of the nearest cgroup with a memory limit (a container run with --memory); else /proc/pressure/memory. Without PSI, as on macOS, nothing is held, and the scheduler says so once.
• Runs can read it. Every run gets ORCA_MEMORY_PRESSURE_FILE, and a step can call orca.memory_pressure() to lower its own parallelism while it is high.

Ops › Control's Memory pressure card shows the scheduler host's reading, and /api/workers each worker's.

Horizontal: more runs, more machines

How many at once

max_runs = "auto" starts another run of the pipeline whenever its request fits in what the host's running runs weren't granted, up to a ceiling of 64 ("auto..N" sets it to N). A bigger host runs more of them at once with no change to orca.toml: at 2 GB a run, 4 on a host with 8 GB to grant and 12 on one with 24. It needs a request to weigh, declared or learned, and starts one run at a time until it has one. On EC2, Kubernetes and remote workers, where there is no single host's room to weigh, the ceiling is the limit, and each worker still admits what it takes against its own host.

From 0.24.0, fill_max_runs = "auto" or "auto..N" does the same for a pipeline's fill runs, with the same ceiling of 64. Before that, fill_max_runs is a count. An older morca reads "auto" as its ceiling.

Pools and autoscaling

When one host is full, place a pipeline on a pool, placement = "worker:<pool>", and add machines. A pool's workers can be started by hand (Remote workers), or morca can grow and shrink the pool itself from a [pools.<name>] table: it makes a Hetzner or EC2 machine while runs wait, up to max_machines, and removes it after idle_seconds with nothing to do (Pools and autoscaling). On a pool, max_runs and fill_max_runs cap the pipeline across all its machines, and each machine's grants cap what runs there.

One run over several machines

From 0.24.0, workers = N on a pipeline placed on a pool lets one run's dista cluster use up to N more machines than the one that starts it. Members join a running run when they have room, and on a pool morca grows, a run that wants more members grows the pool (One run over several machines). On the Kubernetes and EC2 runners each run gets a dista cluster of its own instead, sized by its registration (Runners).

Which knob when

orca.toml has each key, and orca-toml.md the full rules.