# Pools and autoscaling

> Worker pools in morca: workers that claim a pool's runs over HTTPS, pools that morca grows and shrinks itself on Hetzner or EC2, what counts as demand, how a machine boots and is removed, out-of-stock and quota fallback, the private network, and one run spread over several machines with a dista cluster of its own.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

A pool is a set of machines that run the runs of the pipelines placed on it. You can start its workers by hand, or let morca make and remove machines as runs wait. One run can also spread its dista cluster over several machines of a pool.

## Worker pools

A pipeline registered with `placement = "worker:<pool>"` is never started by the scheduler. Its runs stay queued until a worker of that pool claims one over the UI's HTTPS API. The worker runs the pipeline's binary on its own machine and records the run's state through the same API, so it never needs database credentials or an inbound port. A pool's name is 1 to 32 of `a-z`, `0-9`, `-` and `_`.

[Remote workers](https://morca.dev/docs/workers) covers a worker started by hand: its token, the container, fills on a pool and what happens when a worker dies. The rest of this page is about pools that morca runs for you.

## Autoscaling

Since 0.23.0, a `[pools.<name>]` table in orca.toml makes a pool that morca grows and shrinks itself. Pipelines run on it with `placement = "worker:<name>"`, as on workers started by hand. `orca register` records the pool in `orca.pools`, and the leading scheduler looks at every pool every `ORCA_POOL_SCALE_SECONDS` (15).

```
[pools.lab]
provider = "hetzner"
machines = [
  { type = "cax31", wait = 600 },   # prefer this; keep trying it for 10 min
  { type = "cax21", wait = 600 },
  { type = "cpx32" },               # the last: no wait, keep trying it
]
locations = ["hel1", "nbg1", "fsn1"]
max_machines = 3
idle_seconds = 900

[pipelines.history]
placement = "worker:lab"
memory_gb = "auto"
```

| Key | What it sets | Default |
| --- | --- | --- |
| provider | where machines are made: "hetzner" or "ec2" | required |
| machines | machine types in order of preference, each with a wait in seconds; every entry but the last needs one | required |
| locations | where to try each type, in order: Hetzner locations (required there), or EC2 subnet IDs | none |
| max\_machines | the most machines the pool has at once, 1 to 100 | 1 |
| idle\_seconds | how long a machine may sit idle, with nothing waiting, before it's removed | 900 |

[orca.toml](https://morca.dev/docs/config) has every key and its limits. A pool the file no longer has is deleted when you register again, and its machines are removed once they are idle.

### Machine types and waits

The `machines` list is a fallback order. morca tries the type it's on in each location in turn, and the first location that has it gets the machine. If no location has it, morca keeps trying the same type, at most once every `ORCA_POOL_TRY_SECONDS` (60), until that type's `wait` has passed. Then it moves to the next type. The last type has no wait and is tried for as long as there is demand. Once a machine is made, or the demand goes away, morca starts again from the first type. A machine that is running is never replaced by a type higher in the list: the order only matters when a machine is added.

morca also skips a type whose CPU architecture the worker image has no variant for. It reads the image's manifest from the registry (an OCI index or a Docker manifest list, every `ORCA_POOL_IMAGE_SECONDS`, 600) and compares its architectures with the one the provider gives for the type. An arm64-only image never gets an x86 machine, and the pool's status says the type was skipped.

### Out of stock and quota

A provider can refuse a type for two different reasons, and morca treats them differently:

- **Out of stock.** Hetzner's `resource_unavailable`, `placement_error` or `server_type_unavailable`, or EC2's `InsufficientInstanceCapacity`, `InsufficientCapacity` or `Unsupported`, means "not here, not now". morca tries the next location, and keeps trying the type for its `wait`, as above.
- **Quota** (from 0.24.0). Hetzner's `resource_limit_exceeded`, or EC2's `VcpuLimitExceeded` or `InstanceLimitExceeded`, means the project's limit is reached. Waiting doesn't free quota, and every location shares it, so morca moves on at once to the next usable type in `machines`, in the same pass, since a smaller type may still fit. When none fits, the pool's status says `can't grow: the project's quota fits none of its types`.
- **Anything else** is reported in the pool's status and tried again later.

`orca.pools.status` always says what the last try came to.

## Demand

A pool needs another machine when its pipelines (not paused) have runs that have waited `ORCA_POOL_GROW_AFTER` seconds (20) and that a worker could start. That counts no more of a pipeline's runs than its `max_runs` less those running, plus its fill runs. Workers already in the pool take what they have room for first, and what is left waiting is the demand. A running run that wants more cluster members than have joined it (below, from 0.24.0) counts as demand too.

While there is demand, no machine of the pool is still booting, and the pool has fewer than `max_machines`, morca makes one machine. So a pool grows one machine at a time, each once the one before it has joined.

## Making a machine

Each machine is named `orca-pool-<pool>-<n>` (from 0.24.0; `orca-<pool>-<n>` before) and labelled with the pool and this stack's id, so several stacks can share one cloud project and keep their machines apart. morca creates a worker token for the machine alone and puts it in the machine's startup script, which holds no other secret. When the machine boots, the script:

1. asks the UI's `/api/worker/boot` with that token, over TLS, for the rest: the worker image (`ORCA_WORKER_IMAGE`, the server's own), the login to pull it, and the bucket's variables (`ORCA_WAREHOUSE`, `ORCA_LOGS` and the `AWS_*` ones from the UI's environment);
2. finds its private address in the provider's metadata, for runs spread over several machines (from 0.24.0);
3. installs Docker if it isn't there, pulls the image (Docker picks the variant for the machine's architecture from the same manifest), and runs `orca worker --pool <pool> --name <machine> --max-runs 64` with host networking and run cgroups.

`/api/worker/boot` answers only a token the autoscaler made for a machine that is live, so a token from `orca worker-token create` gets nothing from it. The worker takes as many runs as its own host can [grant](https://morca.dev/docs/scaling#vertical), up to 64\.

## Removing machines

To remove a machine, morca revokes its token, deletes the machine, and ends its row in `orca.pool_machines` saying why. That happens when:

- it has run nothing for the pool's `idle_seconds` and the pool has no demand;
- its pool is no longer in orca.toml, once it is idle;
- its worker never showed up within `ORCA_POOL_BOOT_SECONDS` (900), or went silent for `ORCA_POOL_LOST_SECONDS` (300). Its runs are then queued again, as any lost worker's are;
- it carries the pool's and this stack's labels but morca has no live row for it, such as a machine made just before the scheduler stopped.

A machine that runs only a cluster member of another machine's run is busy, not idle.

## Hetzner

The scheduler needs a read-write Hetzner API token for the project the machines go in: `ORCA_HCLOUD_TOKEN`, or `ORCA_HCLOUD_TOKEN_SECRET` naming the secret that holds it. Each machine gets the firewall and SSH keys named in `ORCA_HCLOUD_FIREWALL` and `ORCA_HCLOUD_SSH_KEYS`, and the OS image `ORCA_HCLOUD_IMAGE` (`ubuntu-24.04`).

From 0.24.0, every machine joins a private network, `orca-<stack id>`, which morca makes in `eu-central` over `10.77.0.0/16` if it isn't there (`ORCA_HCLOUD_NETWORK`, `ORCA_HCLOUD_NETWORK_ZONE` and `ORCA_HCLOUD_NETWORK_RANGE` change it; `none` for none). Its worker advertises its address on that network, so a run spread over several machines talks over it.

On the Hetzner stack, `HCLOUD_POOL_TOKEN_FILE` in `hetzner.env` (or a key in `SECRETS_FILE`) is all you set: `deploy.sh` sends the token to the server and fills in the rest, with the stack's own firewall (ssh from `ADMIN_IP` only) and SSH key ([hetzner.md](https://github.com/morca-dev/morca/blob/main/docs/hetzner.md#worker-pools-that-grow-themselves)).

## EC2

With `provider = "ec2"`, the scheduler uses `AWS_REGION` and its credentials or instance role. A pool's `locations` are subnet IDs, tried in order; with none, EC2's default subnet or `ORCA_EC2_SUBNET`. Each instance gets:

- the security group `ORCA_POOL_EC2_SECURITY_GROUP` and instance profile `ORCA_POOL_EC2_PROFILE` (else `ORCA_EC2_SECURITY_GROUP` and `ORCA_EC2_PROFILE`). The profile's role is how its worker reaches the bucket;
- Ubuntu 24.04 for the type's architecture, from Canonical's public SSM parameter, or the image in `ORCA_POOL_EC2_AMI_ARM64` or `ORCA_POOL_EC2_AMI_AMD64`;
- IMDSv2 only, terminate on shutdown, and the tags `orca=pool`, `orca-pool=<pool>`, `orca-owner=<stack>` and `Name`. Its worker advertises the instance's private address.

## One run over several machines

From 0.24.0, a pipeline with dista tasks can spread one run's cluster over several workers of its pool. `workers = N` (or `orca register --workers N`) on a pipeline placed `worker:<pool>` lets each run use up to N more machines than the one that starts it. Without it, a run stays on one machine.

```
[pipelines.fanout]
placement = "worker:lab"
workers = 2      # up to 2 more machines join each run
```

- **The head.** The worker that claims the run starts it as a cluster of its own: a dista head on a free port, a dista worker, and the run's driver. It reports the head's address to the UI. That address is the one the worker was started with, `--advertise <address>` or `ORCA_WORKER_ADVERTISE`.
- **Members.** Another worker of the pool that has room and nothing to start is given the run to join. It starts a dista worker against the head, sized as a run of the pipeline would be on its machine and counted against its room. dista puts a worker that joins a running job to work at once.
- **The cluster token.** Each run start gets a token of its own, given as `DISTA_TOKEN` to the head, the driver and every member, so nothing else on the network can join the run ([Tokens and secrets](https://morca.dev/docs/security#cluster-token)).
- **The end.** When the run ends, each member is told to stop on its worker's next claim, and a member's dista worker gives up on a head that is gone within 10 seconds anyway. `orca.run_members` keeps who joined each run start, with its grant, and when and why it ended.

A worker without `--advertise` neither starts a run as a cluster nor joins one, so its runs run on its machine alone. On a pool morca grows, all of this is set up for you: each machine advertises its private address, its worker uses host networking so dista's ports are on that address, and a run that wants more members grows the pool, up to `max_machines`.

## In the UI

- **Ops › Pools** (`/ops/pools?kind=self-hosted`, `/api/pools`) has a row per worker's machine, autoscaled ones included: its state, image, CPUs, memory, runs and last heartbeat.
- **Autoscaled pools** (from 0.24.0). The same page has a card with a row per `[pools.<name>]`: its provider, its types with their waits, its locations, live machines against `max_machines`, its `idle_seconds`, the type it is trying and what the last try came to. `/api/pools` has them as `autoscaled`.
- **A run's Cluster card** (from 0.24.0). A run spread over several workers lists its head and each member: the worker, the run start, its address, its grant, when it joined and ended, and why. The run's JSON has the same rows.

[workers.md](https://github.com/morca-dev/morca/blob/main/docs/workers.md#one-run-over-several-workers) and [orca-toml.md](https://github.com/morca-dev/morca/blob/main/docs/orca-toml.md#worker-pools) in the morca repository have the full detail and every variable.

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.