# Roadmap

> morca's resource roadmap in two halves, after Google's Autopilot: vertical scaling, the right size for each run (measuring its memory and CPU, admitting it by what it needs, containing an overrun to its own run, back pressure inside it), and horizontal scaling, the right number of runs and machines, with milestones, the issues for each step and the papers behind them.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

The resource plan has two halves, as in Google's Autopilot: vertical scaling gives each run the right size, and horizontal scaling runs the right number of them, on the right number of machines.

## Vertical: the right size for each run

The goal: a run that needs more memory than expected fails alone and is retried bigger, instead of taking other runs down with it, and a pipeline on a new machine uses its cores without hand-set concurrency settings.

"Never out of memory" isn't promised. Some memory use depends on the data, such as a very large repository's clone or an unusually busy hour, and is only known while the step runs. Packing runs onto a machine optimally is a bin-packing problem; the aim is to keep the machine busy without anything being killed.

The host itself is resized by hand: on the Hetzner stack, `rescale.sh up` moves the server from a CAX21 to a CAX41 for an import, and `rescale.sh down` moves it back ([Deploying](https://morca.dev/docs/deploy#rescaling)).

### Four layers

| Layer | What it does | Today (0.24.0) | Next |
| --- | --- | --- | --- |
| Measure | Know what each run actually used | Each run's peak memory, CPU time and peak cores are recorded and shown on the run, its steps and the pipeline's Resources card ([#203](https://github.com/morca-dev/morca/pull/203)). cpus = "auto" and memory\_gb = "auto" take requests learned from those peaks, as Autopilot's vertical recommender does ([#269](https://github.com/morca-dev/morca/issues/269)) | |
| Admit | Start a run only if its request fits what's free | cpus and memory\_gb requests, admitted against the host's capacity less a reserve, with greedy grants passed as ORCA\_GRANT\_\*, on the local runner, control and remote workers ([#208](https://github.com/morca-dev/morca/pull/208)). Nothing new starts while the machine's memory pressure (PSI) is over 10% ([#225](https://github.com/morca-dev/morca/pull/225)) | [morca #171](https://github.com/morca-dev/morca/issues/171) Requests on EC2 and Kubernetes, and a reservation so a large run isn't starved by small ones |
| Contain | An overrun hits only its own run | A cgroup per run, at its grant: throttled there, killed alone 25% above it, retried 1.5× bigger, and the pipeline remembers ([#226](https://github.com/morca-dev/morca/pull/226)). Used wherever morca can make cgroups (on the Hetzner stack, after orca-stack configure CGROUPS=1); otherwise first\_to\_go and the control cap | [morca #239](https://github.com/morca-dev/morca/issues/239) On by default, and each run's own memory pressure next to its limit[morca #229](https://github.com/morca-dev/morca/issues/229) CPU too: runs at a lower priority than the scheduler, the UI and Postgres |
| Back pressure inside a run | A job adapts how much work it keeps in flight | "auto" settings sized from the grant (orca.auto\_count, DISTA\_WORKER\_SLOTS); every run can read the machine's pressure from ORCA\_MEMORY\_PRESSURE\_FILE. dista's queue and store sizes are still hand-set | [dista #123](https://github.com/morca-dev/dista/issues/123) Size in-flight work from the run's grant and live memory |

### Order

Measure (morca #200), grants (#171), containment (#201) and admission by memory pressure (#202) are in 0.15.0, each built on the one before: grants need measurements, containment uses the grants as limits, and admission holds new runs off a machine that is already short of memory. Requests learned from the measured peaks ([orca #269](https://github.com/morca-dev/morca/issues/269)) followed, so a pipeline gets the right size without setting `memory_gb` at all. Next is dista #123, which needs both the grant and the pressure signal.

## Horizontal: the right number

The goal: a pipeline runs as many runs at once, on as many machines, as its queue needs and its pools can give, with no hand-set counts. A backfill on a bigger machine, or on a pool with one more worker, runs more at once without an orca.toml change, and a pool with nothing queued costs nothing.

Where it stands in 0.24.0:

| | Today (0.24.0) |
| --- | --- |
| Inside a run | A run's dista workers, on its machine: DISTA\_WORKER\_SLOTS = "auto" makes them the granted CPUs over the run's local workers |
| Runs at once | max\_runs = "auto" and, from 0.24.0, fill\_max\_runs = "auto": another run starts while its request fits in what's free, up to a ceiling ([Sizing and scaling](https://morca.dev/docs/scaling)) |
| Pools | \[pools.<name>\] with provider = "hetzner" or "ec2" makes machines when runs wait and removes them once idle, trying each type in machines in turn ([Pools and autoscaling](https://morca.dev/docs/pools)). Machines joined by hand still run orca worker --pool ([Remote workers](https://morca.dev/docs/workers)) |
| Across machines | From 0.24.0, workers = N spreads one run's dista cluster over N of a pool's machines, on a private network, with a token of its own |
| Kubernetes and EC2 runners | A dista cluster of its own per run, created and deleted with it; on EC2, \--workers MIN..MAX adds and drains workers as the head's demand changes ([On Kubernetes and EC2](https://morca.dev/docs/run#runners)). Kubernetes runs don't get their secrets yet ([#206](https://github.com/morca-dev/morca/issues/206)) |

### Milestones

H1, H3 and H4 are done; H2 is next.

- **H1\. Across runs** ([morca #266](https://github.com/morca-dev/morca/issues/266), done in 0.22.0; fills from 0.24.0). `max_runs = "auto"`: another run of the pipeline starts while its request fits in what's free, with the same admission as a single run, up to a ceiling. `fill_max_runs = "auto"` does the same for fill runs.
- **H3\. Pools that add and remove machines** ([morca #267](https://github.com/morca-dev/morca/issues/267), done in 0.23.0; quota fallback from 0.24.0). A pool lists machine types, each with how long to wait for it. When runs wait for the pool and don't fit on its live workers, morca makes a machine that boots into a worker of the pool, with the image for its CPU architecture; an idle one is removed. Hetzner Cloud and EC2; a type out of stock or over the project's quota is passed over for the next.
- **H4\. One run across several machines** ([morca #268](https://github.com/morca-dev/morca/issues/268), from 0.24.0). `workers = N`: a run on a pool gets workers on N of its machines, and its dista cluster spans them over a private network, with results identical to one machine. From 0.24.0 a run's page shows the machines it spread over.
- **H2\. Machines joined by hand** ([morca #220](https://github.com/morca-dev/morca/issues/220), next). `orca host join` makes a machine you already have a worker of a pool once: it runs the worker at the server's image tag, upgrades after a deploy, reports the machine and drains on request. The worker follow-ups in [#177](https://github.com/morca-dev/morca/issues/177) come with it.
- **H5\. More providers, and jobs that grow** ([morca #263](https://github.com/morca-dev/morca/issues/263), later). Pools on GCP ([#263](https://github.com/morca-dev/morca/issues/263)) and Kubernetes ([#206](https://github.com/morca-dev/morca/issues/206)), and a running job that grows as the pool adds machines ([dista #76](https://github.com/morca-dev/dista/issues/76)).

## Prior art

What each of these gives the plan, and which half it informs:

- Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes, "Large-scale cluster management at Google with Borg", EuroSys 2015\. _Vertical:_ requests against usage, and limits. [Link](https://doi.org/10.1145/2741948.2741964)
- Krzysztof Rzadca et al., "Autopilot: workload autoscaling at Google", EuroSys 2020\. _Both:_ the split into vertical and horizontal scaling, and limits set from usage history, which is what "auto" means here. [Link](https://doi.org/10.1145/3342195.3387524)
- Christina Delimitrou and Christos Kozyrakis, "Quasar: resource-efficient and QoS-aware cluster management", ASPLOS 2014\. _Vertical:_ learning what a job needs instead of trusting its declaration. [Link](https://doi.org/10.1145/2541940.2541941)
- Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker and Ion Stoica, "Dominant Resource Fairness: fair allocation of multiple resource types", NSDI 2011\. _Both:_ admitting with CPU and memory together, for one run and for how many fit. [Link](https://www.usenix.org/conference/nsdi11/dominant-resource-fairness-fair-allocation-multiple-resource-types)
- Robert Grandl, Ganesh Ananthanarayanan, Srikanth Kandula, Sriram Rao and Aditya Akella, "Multi-resource packing for cluster schedulers" (Tetris), SIGCOMM 2014\. _Horizontal:_ the same, as packing runs onto machines. [Link](https://doi.org/10.1145/2619239.2626334)
- Matt Welsh, David Culler and Eric Brewer, "SEDA: an architecture for well-conditioned, scalable internet services", SOSP 2001\. _Vertical:_ admission control and bounded queues. [Link](https://doi.org/10.1145/502034.502057)
- Kathleen Nichols and Van Jacobson, "Controlling queue delay", ACM Queue, 2012\. _Vertical:_ sizing in-flight work from measured delay. [Link](https://doi.org/10.1145/2208917.2209336)
- Johannes Weiner et al., "TMO: transparent memory offloading in datacenters", ASPLOS 2022\. _Vertical:_ memory pressure (PSI) as the signal. [Link](https://doi.org/10.1145/3503222.3507731)
- Viktor Leis, Peter Boncz, Alfons Kemper and Thomas Neumann, "Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age", SIGMOD 2014\. _Both:_ work in small units sized to the cores and memory available, on one machine or spread over several. [Link](https://doi.org/10.1145/2588555.2610507)

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.