# Remote workers

> Running some of morca's pipelines on a second machine: what a remote worker is, a worker token, a placement, the worker container that needs only outbound HTTPS, fills on a pool, what happens when a worker dies, deploys, the census as the example, and the limits.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

A remote worker runs a pipeline's runs on a second machine. It needs no database access and no inbound ports, only outbound HTTPS to the morca UI and to the bucket.

## What it is

The scheduler stays the only brain: it fires, senses, queues and retries, and keeps all state in its Postgres. A pipeline registered with `placement = "worker:<pool>"` is never started by the scheduler. Its runs wait, queued, until a worker of that pool claims one. The worker polls the UI's HTTPS API (`/api/worker/*`), runs the pipeline's binary on its own machine, and the run records its state through the same API instead of Postgres. Data and logs go from the worker straight to the warehouse and logs buckets.

That is the reason to use one: a second machine with spare CPU, memory or disk without exposing Postgres to it, and the way to scale out when the first machine is full. A worker holds a token scoped to its pool and nothing else. Since 0.15.0 a run whose worker is gone (not seen for half `ORCA_STALE_SECONDS`, or restarted without it) is queued again for the pool whatever started it, within its `run_retries` starts, and resumes after its last step done; one claimed and never started gets its start back.

The only path from the worker to the server is HTTPS to the UI (claims, state calls, heartbeats). Postgres is never reachable from the worker.

## Setting one up

Deploy the server first. Then four steps: a token, a placement, the worker, and a look at Ops.

### 1\. A token, on the server

```
orca worker-token create --pool bgent --label bgent   # prints the token, once
orca worker-token list
orca worker-token revoke 3
```

The token is scoped to one pool and expires (by default after a year at most). The server keeps only its hash. Copy it to the worker's machine as a file.

### 2\. A placement, in orca.toml

```
[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }
```

Or `orca register --placement worker:bgent`. Register again after the change. The pipeline's page then says its runs are waiting for a worker of the pool.

### 3\. The worker

The simplest worker is the server's own image at the same tag, so the pipeline binaries are at the paths they were registered with. It needs the pool, the UI's URL, the token file and the bucket's settings:

```
docker run -d --name orca-worker --restart always \
  --env-file s3.env -v orca-worker-secrets:/etc/orca/w:ro \
  -e ORCA_WAREHOUSE=s3://<bucket>/warehouse -e ORCA_LOGS=s3://<bucket>/logs \
  -e AWS_REGION=<location> -e AWS_ENDPOINT_URL_S3=https://<location>.your-objectstorage.com \
  ghcr.io/magmalake/orca:<tag> worker --pool bgent --max-runs 3 \
    --remote https://orca.example.com --token-file /etc/orca/w/token
```

`--max-runs` is how many runs the worker takes at once. A worker with `ORCA_DATABASE_URL` set refuses to start. A worker admits the runs of pipelines with a [request](https://morca.dev/docs/run#resources) against its own host, so its log says what it can grant: `grants up to 8 CPUs and 11421 MB`, less `ORCA_RESERVE_CPUS` and `ORCA_RESERVE_MEMORY_MB` (0 and 512 on a worker). Since 0.15.0 it also starts no new run while its own host's memory pressure is over `ORCA_ADMIT_MEMORY_PRESSURE` (a waiting run says `... on worker <name>`), gives each run a cgroup with a memory limit where it can make one, and samples the machine beside each run, so the run's page has its Machines card.

### 4\. Check Ops

Ops › Pools (`/ops/pools?kind=self-hosted`, and `/api/pools` as JSON) has a row per worker token's machine, in pool `worker:<pool>`: its state (`live`, or `gone`), image, CPUs, memory, current runs and last heartbeat, and its token and label. The machine's page lists the runs it claimed. `/api/workers` is still each token as JSON. A run's page says which worker ran it. `docker logs orca-worker` shows each run it starts and how it exited.

## Fills on a pool

Since 0.15.0 a pipeline placed on a pool with `fills = "all"` fills its range as a local one does: the scheduler queues its fill runs in the same order and window, and a worker of the pool claims them. `fill_max_runs` caps the pipeline's fill runs across the whole pool, queued ones included, and each worker's `--max-runs` and grants cap one machine. A worker's machine runs anyway, so a pool's fill runs cost nothing against `fill_budget_per_day` ([workers.md](https://github.com/morca-dev/morca/blob/main/docs/workers.md#fills-on-a-pool)).

## The census

[The census](https://github.com/morca-dev/census) runs on a Hetzner CAX21: the scheduler, the UI, Postgres, the site, and gharchive's fill, mojo\_repos and mojo\_stats. The git-heavy pipelines, mojo\_history, mojo\_versions and mojo\_activity, run on a second machine, bgent, an Apple-silicon Mac running Docker. It is arm64 like the CAX21, so it runs the same image. Each of the three is registered like this:

```
[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }
```

The worker is the container above, with the census's bucket and one more setting, `MOJO_HISTORY_WORKDIR=/work/clones` on a volume, for the clones. The [census README](https://github.com/morca-dev/census#a-second-machine) has the exact commands.

## Limits

- **The same tag.** The worker runs the code of its own image. Deploy the server first, then restart the worker on the same tag.
- **Secrets.** A pipeline with secrets needs `ORCA_SECRET_FETCH` on the worker, to read them there. The spec names the secrets and never carries their values.
- **Machines you already have.** On these the worker is started by hand, as above; `orca host join` is [morca #220](https://github.com/morca-dev/morca/issues/220). A pool whose machines morca makes and removes on Hetzner Cloud or EC2 needs none of this: see [Pools and autoscaling](https://morca.dev/docs/pools).
- **Deploys.** Since 0.18.0 a server deploy doesn't wait for a worker's runs, and the UI is replaced without downtime, so the worker's claims, state calls and heartbeats go on to the new UI. `rescale.sh` does wait for them, since the UI goes down with the server ([Deploying](https://morca.dev/docs/deploy)).
- **The UI's port.** By default the worker API shares the UI's port, so a slow page can delay a worker. From morca 0.15.0, `ORCA_WORKER_ADDR` serves it on a listener of its own.

## Docker Desktop and the token

Docker Desktop runs containers in a virtual machine, which may not see a file bind-mounted from under `~/.config`: the mount shows up as an empty directory, and the worker finds no token. Put the token in a Docker volume instead:

```
docker run --rm -i --entrypoint sh -v orca-worker-secrets:/s \
  ghcr.io/magmalake/orca:<tag> -c 'cat > /s/token && chmod 600 /s/token' < worker-token
```

The worker container above mounts that volume read-only at `/etc/orca/w`.

## More

[workers.md](https://github.com/morca-dev/morca/blob/main/docs/workers.md) in the morca repository has the flags, how a dead worker's runs are queued again, the state calls and the network details. [Scheduling and running](https://morca.dev/docs/run#placement) says where placement fits among the runners: on [Kubernetes and EC2](https://morca.dev/docs/run#runners) each run gets a Dista cluster of its own instead, created and deleted with it. The [roadmap](https://morca.dev/docs/roadmap#horizontal) has what comes next for pools.

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.