Docs
Roadmap
The resource plan has two halves, as in Google's Autopilot: vertical scaling gives each run the right size, and horizontal scaling runs the right number of them, on the right number of machines.
Vertical: the right size for each run
The goal: a run that needs more memory than expected fails alone and is retried bigger, instead of taking other runs down with it, and a pipeline on a new machine uses its cores without hand-set concurrency settings.
"Never out of memory" isn't promised. Some memory use depends on the data, such as a very large repository's clone or an unusually busy hour, and is only known while the step runs. Packing runs onto a machine optimally is a bin-packing problem; the aim is to keep the machine busy without anything being killed.
The host itself is resized by hand: on the Hetzner stack, rescale.sh up
moves the server from a CAX21 to a CAX41 for an import, and rescale.sh down
moves it back (Deploying).
Four layers
| Layer | What it does | Today (0.24.0) | Next |
|---|---|---|---|
| Measure | Know what each run actually used | Each run's peak memory, CPU time and peak cores are recorded and shown on the run, its steps and the pipeline's Resources card (#203). cpus = "auto" and memory_gb = "auto" take requests learned from those peaks, as Autopilot's vertical recommender does (#269) | |
| Admit | Start a run only if its request fits what's free | cpus and memory_gb requests, admitted against the host's capacity less a reserve, with greedy grants passed as ORCA_GRANT_*, on the local runner, control and remote workers (#208). Nothing new starts while the machine's memory pressure (PSI) is over 10% (#225) | morca #171 Requests on EC2 and Kubernetes, and a reservation so a large run isn't starved by small ones |
| Contain | An overrun hits only its own run | A cgroup per run, at its grant: throttled there, killed alone 25% above it, retried 1.5× bigger, and the pipeline remembers (#226). Used wherever morca can make cgroups (on the Hetzner stack, after orca-stack configure CGROUPS=1); otherwise first_to_go and the control cap | morca #239 On by default, and each run's own memory pressure next to its limit morca #229 CPU too: runs at a lower priority than the scheduler, the UI and Postgres |
| Back pressure inside a run | A job adapts how much work it keeps in flight | "auto" settings sized from the grant (orca.auto_count, DISTA_WORKER_SLOTS); every run can read the machine's pressure from ORCA_MEMORY_PRESSURE_FILE. dista's queue and store sizes are still hand-set | dista #123 Size in-flight work from the run's grant and live memory |
Order
Measure (morca #200), grants (#171), containment (#201) and admission by memory pressure
(#202) are in 0.15.0, each built on the one before: grants need measurements,
containment uses the grants as limits, and admission holds new runs off a machine that
is already short of memory. Requests learned from the measured peaks
(orca #269) followed, so a pipeline
gets the right size without setting memory_gb at all. Next is dista #123,
which needs both the grant and the pressure signal.
Horizontal: the right number
The goal: a pipeline runs as many runs at once, on as many machines, as its queue needs and its pools can give, with no hand-set counts. A backfill on a bigger machine, or on a pool with one more worker, runs more at once without an orca.toml change, and a pool with nothing queued costs nothing.
Where it stands in 0.24.0:
| Today (0.24.0) | |
|---|---|
| Inside a run | A run's dista workers, on its machine: DISTA_WORKER_SLOTS = "auto" makes them the granted CPUs over the run's local workers |
| Runs at once | max_runs = "auto" and, from 0.24.0, fill_max_runs = "auto": another run starts while its request fits in what's free, up to a ceiling (Sizing and scaling) |
| Pools | [pools.<name>] with provider = "hetzner" or "ec2" makes machines when runs wait and removes them once idle, trying each type in machines in turn (Pools and autoscaling). Machines joined by hand still run orca worker --pool (Remote workers) |
| Across machines | From 0.24.0, workers = N spreads one run's dista cluster over N of a pool's machines, on a private network, with a token of its own |
| Kubernetes and EC2 runners | A dista cluster of its own per run, created and deleted with it; on EC2, --workers MIN..MAX adds and drains workers as the head's demand changes (On Kubernetes and EC2). Kubernetes runs don't get their secrets yet (#206) |
Milestones
H1, H3 and H4 are done; H2 is next.
- H1. Across runs (morca #266, done in 0.22.0; fills from 0.24.0).
max_runs = "auto": another run of the pipeline starts while its request fits in what's free, with the same admission as a single run, up to a ceiling.fill_max_runs = "auto"does the same for fill runs. - H3. Pools that add and remove machines (morca #267, done in 0.23.0; quota fallback from 0.24.0). A pool lists machine types, each with how long to wait for it. When runs wait for the pool and don't fit on its live workers, morca makes a machine that boots into a worker of the pool, with the image for its CPU architecture; an idle one is removed. Hetzner Cloud and EC2; a type out of stock or over the project's quota is passed over for the next.
- H4. One run across several machines (morca #268, from 0.24.0).
workers = N: a run on a pool gets workers on N of its machines, and its dista cluster spans them over a private network, with results identical to one machine. From 0.24.0 a run's page shows the machines it spread over. - H2. Machines joined by hand (morca #220, next).
orca host joinmakes a machine you already have a worker of a pool once: it runs the worker at the server's image tag, upgrades after a deploy, reports the machine and drains on request. The worker follow-ups in #177 come with it. - H5. More providers, and jobs that grow (morca #263, later). Pools on GCP (#263) and Kubernetes (#206), and a running job that grows as the pool adds machines (dista #76).
Prior art
What each of these gives the plan, and which half it informs:
- Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes, "Large-scale cluster management at Google with Borg", EuroSys 2015. Vertical: requests against usage, and limits. Link
- Krzysztof Rzadca et al., "Autopilot: workload autoscaling at Google", EuroSys 2020. Both: the split into vertical and horizontal scaling, and limits set from usage history, which is what "auto" means here. Link
- Christina Delimitrou and Christos Kozyrakis, "Quasar: resource-efficient and QoS-aware cluster management", ASPLOS 2014. Vertical: learning what a job needs instead of trusting its declaration. Link
- Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker and Ion Stoica, "Dominant Resource Fairness: fair allocation of multiple resource types", NSDI 2011. Both: admitting with CPU and memory together, for one run and for how many fit. Link
- Robert Grandl, Ganesh Ananthanarayanan, Srikanth Kandula, Sriram Rao and Aditya Akella, "Multi-resource packing for cluster schedulers" (Tetris), SIGCOMM 2014. Horizontal: the same, as packing runs onto machines. Link
- Matt Welsh, David Culler and Eric Brewer, "SEDA: an architecture for well-conditioned, scalable internet services", SOSP 2001. Vertical: admission control and bounded queues. Link
- Kathleen Nichols and Van Jacobson, "Controlling queue delay", ACM Queue, 2012. Vertical: sizing in-flight work from measured delay. Link
- Johannes Weiner et al., "TMO: transparent memory offloading in datacenters", ASPLOS 2022. Vertical: memory pressure (PSI) as the signal. Link
- Viktor Leis, Peter Boncz, Alfons Kemper and Thomas Neumann, "Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age", SIGMOD 2014. Both: work in small units sized to the cores and memory available, on one machine or spread over several. Link