morca

Docs

Remote workers

A remote worker runs a pipeline's runs on a second machine. It needs no database access and no inbound ports, only outbound HTTPS to the morca UI and to the bucket.

What it is

The scheduler stays the only brain: it fires, senses, queues and retries, and keeps all state in its Postgres. A pipeline registered with placement = "worker:<pool>" is never started by the scheduler. Its runs wait, queued, until a worker of that pool claims one. The worker polls the UI's HTTPS API (/api/worker/*), runs the pipeline's binary on its own machine, and the run records its state through the same API instead of Postgres. Data and logs go from the worker straight to the warehouse and logs buckets.

That is the reason to use one: a second machine with spare CPU, memory or disk without exposing Postgres to it, and the way to scale out when the first machine is full. A worker holds a token scoped to its pool and nothing else. Since 0.15.0 a run whose worker is gone (not seen for half ORCA_STALE_SECONDS, or restarted without it) is queued again for the pool whatever started it, within its run_retries starts, and resumes after its last step done; one claimed and never started gets its start back.

A remote worker and the server The worker polls the UI over HTTPS to claim runs and record their state; the UI and scheduler use Postgres. Server and worker both read and write the bucket directly. server scheduler orca ui Postgres worker orca worker --pool bgent runs the pipeline's binary HTTPS bucket: warehouse and logs
The only path from the worker to the server is HTTPS to the UI (claims, state calls, heartbeats). Postgres is never reachable from the worker.

Setting one up

Deploy the server first. Then four steps: a token, a placement, the worker, and a look at Ops.

1. A token, on the server

orca worker-token create --pool bgent --label bgent   # prints the token, once
orca worker-token list
orca worker-token revoke 3

The token is scoped to one pool and expires (by default after a year at most). The server keeps only its hash. Copy it to the worker's machine as a file.

2. A placement, in orca.toml

[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }

Or orca register --placement worker:bgent. Register again after the change. The pipeline's page then says its runs are waiting for a worker of the pool.

3. The worker

The simplest worker is the server's own image at the same tag, so the pipeline binaries are at the paths they were registered with. It needs the pool, the UI's URL, the token file and the bucket's settings: 

docker run -d --name orca-worker --restart always \
  --env-file s3.env -v orca-worker-secrets:/etc/orca/w:ro \
  -e ORCA_WAREHOUSE=s3://<bucket>/warehouse -e ORCA_LOGS=s3://<bucket>/logs \
  -e AWS_REGION=<location> -e AWS_ENDPOINT_URL_S3=https://<location>.your-objectstorage.com \
  ghcr.io/magmalake/orca:<tag> worker --pool bgent --max-runs 3 \
    --remote https://orca.example.com --token-file /etc/orca/w/token

--max-runs is how many runs the worker takes at once. A worker with ORCA_DATABASE_URL set refuses to start. A worker admits the runs of pipelines with a request against its own host, so its log says what it can grant: grants up to 8 CPUs and 11421 MB, less ORCA_RESERVE_CPUS and ORCA_RESERVE_MEMORY_MB (0 and 512 on a worker). Since 0.15.0 it also starts no new run while its own host's memory pressure is over ORCA_ADMIT_MEMORY_PRESSURE (a waiting run says ... on worker <name>), gives each run a cgroup with a memory limit where it can make one, and samples the machine beside each run, so the run's page has its Machines card.

4. Check Ops

Ops › Pools (/ops/pools?kind=self-hosted, and /api/pools as JSON) has a row per worker token's machine, in pool worker:<pool>: its state (live, or gone), image, CPUs, memory, current runs and last heartbeat, and its token and label. The machine's page lists the runs it claimed. /api/workers is still each token as JSON. A run's page says which worker ran it. docker logs orca-worker shows each run it starts and how it exited.

Fills on a pool

Since 0.15.0 a pipeline placed on a pool with fills = "all" fills its range as a local one does: the scheduler queues its fill runs in the same order and window, and a worker of the pool claims them. fill_max_runs caps the pipeline's fill runs across the whole pool, queued ones included, and each worker's --max-runs and grants cap one machine. A worker's machine runs anyway, so a pool's fill runs cost nothing against fill_budget_per_day (workers.md).

The census

The census runs on a Hetzner CAX21: the scheduler, the UI, Postgres, the site, and gharchive's fill, mojo_repos and mojo_stats. The git-heavy pipelines, mojo_history, mojo_versions and mojo_activity, run on a second machine, bgent, an Apple-silicon Mac running Docker. It is arm64 like the CAX21, so it runs the same image. Each of the three is registered like this: 

[pipelines.mojo_history]
placement = "worker:bgent"
env = { DISTA_WORKER_SLOTS = "3" }

The worker is the container above, with the census's bucket and one more setting, MOJO_HISTORY_WORKDIR=/work/clones on a volume, for the clones. The census README has the exact commands.

Limits

  • The same tag. The worker runs the code of its own image. Deploy the server first, then restart the worker on the same tag.
  • Secrets. A pipeline with secrets needs ORCA_SECRET_FETCH on the worker, to read them there. The spec names the secrets and never carries their values.
  • Machines you already have. On these the worker is started by hand, as above; orca host join is morca #220. A pool whose machines morca makes and removes on Hetzner Cloud or EC2 needs none of this: see Pools and autoscaling.
  • Deploys. Since 0.18.0 a server deploy doesn't wait for a worker's runs, and the UI is replaced without downtime, so the worker's claims, state calls and heartbeats go on to the new UI. rescale.sh does wait for them, since the UI goes down with the server (Deploying).
  • The UI's port. By default the worker API shares the UI's port, so a slow page can delay a worker. From morca 0.15.0, ORCA_WORKER_ADDR serves it on a listener of its own.

Docker Desktop and the token

Docker Desktop runs containers in a virtual machine, which may not see a file bind-mounted from under ~/.config: the mount shows up as an empty directory, and the worker finds no token. Put the token in a Docker volume instead: 

docker run --rm -i --entrypoint sh -v orca-worker-secrets:/s \
  ghcr.io/magmalake/orca:<tag> -c 'cat > /s/token && chmod 600 /s/token' < worker-token

The worker container above mounts that volume read-only at /etc/orca/w.

More

workers.md in the morca repository has the flags, how a dead worker's runs are queued again, the state calls and the network details. Scheduling and running says where placement fits among the runners: on Kubernetes and EC2 each run gets a Dista cluster of its own instead, created and deleted with it. The roadmap has what comes next for pools.