Operations

Pools and autoscaling

A pool is a set of machines that run the runs of the pipelines placed on it. You can start its workers by hand, or let morca make and remove machines as runs wait. One run can also spread its dista cluster over several machines of a pool.

Worker pools

A pipeline registered with placement = "worker:<pool>" is never started by the scheduler. Its runs stay queued until a worker of that pool claims one over the UI's HTTPS API. The worker runs the pipeline's binary on its own machine and records the run's state through the same API, so it never needs database credentials or an inbound port. A pool's name is 1 to 32 of a-z, 0-9, - and _.

Remote workers covers a worker started by hand: its token, the container, fills on a pool and what happens when a worker dies. The rest of this page is about pools that morca runs for you.

Autoscaling

Since 0.23.0, a [pools.<name>] table in orca.toml makes a pool that morca grows and shrinks itself. Pipelines run on it with placement = "worker:<name>", as on workers started by hand. orca register records the pool in orca.pools, and the leading scheduler looks at every pool every ORCA_POOL_SCALE_SECONDS (15).

[pools.lab]
provider = "hetzner"
machines = [
  { type = "cax31", wait = 600 },   # prefer this; keep trying it for 10 min
  { type = "cax21", wait = 600 },
  { type = "cpx32" },               # the last: no wait, keep trying it
]
locations = ["hel1", "nbg1", "fsn1"]
max_machines = 3
idle_seconds = 900

[pipelines.history]
placement = "worker:lab"
memory_gb = "auto"

orca.toml has every key and its limits. A pool the file no longer has is deleted when you register again, and its machines are removed once they are idle.

Machine types and waits

The machines list is a fallback order. morca tries the type it's on in each location in turn, and the first location that has it gets the machine. If no location has it, morca keeps trying the same type, at most once every ORCA_POOL_TRY_SECONDS (60), until that type's wait has passed. Then it moves to the next type. The last type has no wait and is tried for as long as there is demand. Once a machine is made, or the demand goes away, morca starts again from the first type. A machine that is running is never replaced by a type higher in the list: the order only matters when a machine is added.

morca also skips a type whose CPU architecture the worker image has no variant for. It reads the image's manifest from the registry (an OCI index or a Docker manifest list, every ORCA_POOL_IMAGE_SECONDS, 600) and compares its architectures with the one the provider gives for the type. An arm64-only image never gets an x86 machine, and the pool's status says the type was skipped.

Out of stock and quota

A provider can refuse a type for two different reasons, and morca treats them differently:

• Out of stock. Hetzner's resource_unavailable, placement_error or server_type_unavailable, or EC2's InsufficientInstanceCapacity, InsufficientCapacity or Unsupported, means "not here, not now". morca tries the next location, and keeps trying the type for its wait, as above.
• Quota (from 0.24.0). Hetzner's resource_limit_exceeded, or EC2's VcpuLimitExceeded or InstanceLimitExceeded, means the project's limit is reached. Waiting doesn't free quota, and every location shares it, so morca moves on at once to the next usable type in machines, in the same pass, since a smaller type may still fit. When none fits, the pool's status says can't grow: the project's quota fits none of its types.
• Anything else is reported in the pool's status and tried again later.

orca.pools.status always says what the last try came to.

Demand

A pool needs another machine when its pipelines (not paused) have runs that have waited ORCA_POOL_GROW_AFTER seconds (20) and that a worker could start. That counts no more of a pipeline's runs than its max_runs less those running, plus its fill runs. Workers already in the pool take what they have room for first, and what is left waiting is the demand. A running run that wants more cluster members than have joined it (below, from 0.24.0) counts as demand too.

While there is demand, no machine of the pool is still booting, and the pool has fewer than max_machines, morca makes one machine. So a pool grows one machine at a time, each once the one before it has joined.

Making a machine

Each machine is named orca-pool-<pool>-<n> (from 0.24.0; orca-<pool>-<n> before) and labelled with the pool and this stack's id, so several stacks can share one cloud project and keep their machines apart. morca creates a worker token for the machine alone and puts it in the machine's startup script, which holds no other secret. When the machine boots, the script:

1. asks the UI's /api/worker/boot with that token, over TLS, for the rest: the worker image (ORCA_WORKER_IMAGE, the server's own), the login to pull it, and the bucket's variables (ORCA_WAREHOUSE, ORCA_LOGS and the AWS_* ones from the UI's environment);
2. finds its private address in the provider's metadata, for runs spread over several machines (from 0.24.0);
3. installs Docker if it isn't there, pulls the image (Docker picks the variant for the machine's architecture from the same manifest), and runs orca worker --pool <pool> --name <machine> --max-runs 64 with host networking and run cgroups.

/api/worker/boot answers only a token the autoscaler made for a machine that is live, so a token from orca worker-token create gets nothing from it. The worker takes as many runs as its own host can grant, up to 64.

Removing machines

To remove a machine, morca revokes its token, deletes the machine, and ends its row in orca.pool_machines saying why. That happens when:

• it has run nothing for the pool's idle_seconds and the pool has no demand;
• its pool is no longer in orca.toml, once it is idle;
• its worker never showed up within ORCA_POOL_BOOT_SECONDS (900), or went silent for ORCA_POOL_LOST_SECONDS (300). Its runs are then queued again, as any lost worker's are;
• it carries the pool's and this stack's labels but morca has no live row for it, such as a machine made just before the scheduler stopped.

A machine that runs only a cluster member of another machine's run is busy, not idle.

Hetzner

The scheduler needs a read-write Hetzner API token for the project the machines go in: ORCA_HCLOUD_TOKEN, or ORCA_HCLOUD_TOKEN_SECRET naming the secret that holds it. Each machine gets the firewall and SSH keys named in ORCA_HCLOUD_FIREWALL and ORCA_HCLOUD_SSH_KEYS, and the OS image ORCA_HCLOUD_IMAGE (ubuntu-24.04).

From 0.24.0, every machine joins a private network, orca-<stack id>, which morca makes in eu-central over 10.77.0.0/16 if it isn't there (ORCA_HCLOUD_NETWORK, ORCA_HCLOUD_NETWORK_ZONE and ORCA_HCLOUD_NETWORK_RANGE change it; none for none). Its worker advertises its address on that network, so a run spread over several machines talks over it.

On the Hetzner stack, HCLOUD_POOL_TOKEN_FILE in hetzner.env (or a key in SECRETS_FILE) is all you set: deploy.sh sends the token to the server and fills in the rest, with the stack's own firewall (ssh from ADMIN_IP only) and SSH key (hetzner.md).

EC2

With provider = "ec2", the scheduler uses AWS_REGION and its credentials or instance role. A pool's locations are subnet IDs, tried in order; with none, EC2's default subnet or ORCA_EC2_SUBNET. Each instance gets:

• the security group ORCA_POOL_EC2_SECURITY_GROUP and instance profile ORCA_POOL_EC2_PROFILE (else ORCA_EC2_SECURITY_GROUP and ORCA_EC2_PROFILE). The profile's role is how its worker reaches the bucket;
• Ubuntu 24.04 for the type's architecture, from Canonical's public SSM parameter, or the image in ORCA_POOL_EC2_AMI_ARM64 or ORCA_POOL_EC2_AMI_AMD64;
• IMDSv2 only, terminate on shutdown, and the tags orca=pool, orca-pool=<pool>, orca-owner=<stack> and Name. Its worker advertises the instance's private address.

One run over several machines

From 0.24.0, a pipeline with dista tasks can spread one run's cluster over several workers of its pool. workers = N (or orca register --workers N) on a pipeline placed worker:<pool> lets each run use up to N more machines than the one that starts it. Without it, a run stays on one machine.

[pipelines.fanout]
placement = "worker:lab"
workers = 2      # up to 2 more machines join each run

• The head. The worker that claims the run starts it as a cluster of its own: a dista head on a free port, a dista worker, and the run's driver. It reports the head's address to the UI. That address is the one the worker was started with, --advertise <address> or ORCA_WORKER_ADVERTISE.
• Members. Another worker of the pool that has room and nothing to start is given the run to join. It starts a dista worker against the head, sized as a run of the pipeline would be on its machine and counted against its room. dista puts a worker that joins a running job to work at once.
• The cluster token. Each run start gets a token of its own, given as DISTA_TOKEN to the head, the driver and every member, so nothing else on the network can join the run (Tokens and secrets).
• The end. When the run ends, each member is told to stop on its worker's next claim, and a member's dista worker gives up on a head that is gone within 10 seconds anyway. orca.run_members keeps who joined each run start, with its grant, and when and why it ended.

A worker without --advertise neither starts a run as a cluster nor joins one, so its runs run on its machine alone. On a pool morca grows, all of this is set up for you: each machine advertises its private address, its worker uses host networking so dista's ports are on that address, and a run that wants more members grows the pool, up to max_machines.

In the UI

• Ops › Pools (/ops/pools?kind=self-hosted, /api/pools) has a row per worker's machine, autoscaled ones included: its state, image, CPUs, memory, runs and last heartbeat.
• Autoscaled pools (from 0.24.0). The same page has a card with a row per [pools.<name>]: its provider, its types with their waits, its locations, live machines against max_machines, its idle_seconds, the type it is trying and what the last try came to. /api/pools has them as autoscaled.
• A run's Cluster card (from 0.24.0). A run spread over several workers lists its head and each member: the worker, the run start, its address, its grant, when it joined and ended, and why. The run's JSON has the same rows.

workers.md and orca-toml.md in the morca repository have the full detail and every variable.