# orca.worker

> A module of morca's API.

`orca worker`: runs a pool's runs on this machine, talking only to `orca ui`'s API.

```
ORCA_WORKER_TOKEN=orca_... orca worker --pool bgent --remote https://orca.mlake.dev
```

Every `--poll-seconds` (2) the worker posts its heartbeat to
`/api/worker/claim` (`orca.workers`), its name, image, CPUs, memory and
the runs it has, and, while it has fewer than `--max-runs` (1), asks for
one. A run it's given starts as the local runner starts one: the
pipeline's binary (the path it was registered with, which the worker's
image has too) with `resume <run>`, in a process group of its own, its
output staged under `--dir` (`$TMPDIR/orca-worker`), with the pipeline's
`env`, its secrets read here with `ORCA_SECRET_FETCH` and named in
`ORCA_SECRETS`, and `ORCA_STATE_URL` and `ORCA_STATE_TOKEN` in place of a
database: the run records itself over the API (`orca.http_store`). Its
data and logs go where the worker's own `ORCA_WAREHOUSE` and `ORCA_LOGS`
say, as any run's do. When the process ends, its output is appended to
the run's `driver.log` by `<binary> keep-log`, which also fails the
attempts a crash left running; a process that ended before claiming its
run has the run failed, saying why (`/api/worker/exited`). Beside each
run's process it starts `<binary> sample` (`orca.sample`, unless
`ORCA_SAMPLE_SECONDS=0`), which writes this machine's metrics to
`<ORCA_LOGS>/<pipeline>/<run>/metrics/worker-<name>.jsonl`, so the run's
page has its Machines card, as a local run's does. Each poll the
worker samples its runs' process trees, as the scheduler samples a local
run's, and sends the samples with `/api/worker/exited`, so their attempts
get their peak memory and CPU (`orca.usage`). Where the worker can make
cgroups (`orca.cgroup`), each run gets one at its grant, and an
out-of-memory kill is reported with `/api/worker/exited`, which fails the
attempts saying so and raises what the pipeline asks for.

**No database.** The worker refuses to start with `ORCA_DATABASE_URL` set:
nothing on a worker holds database credentials. Its token is
`ORCA_WORKER_TOKEN`, or the first line of `--token-file`; one that's
revoked, expired or for another pool than `--pool` is refused, and the
worker exits (code 3).

**Versions.** The worker's image is `--image`, else `ORCA_IMAGE`. A run it's
given whose pipeline's image isn't that is put back (`/api/worker/release`)
rather than started, though the UI shouldn't hand it one
(`orca.workers.version_refusal`).

**Dying.** A worker that's killed takes its runs with it: each run's
heartbeat thread exits its process once its parent isn't the worker
(`ORCA_WORKER_PID`), so the run's heartbeat goes stale and the scheduler
queues it again, for this worker or another, whatever its trigger
(`orca.scheduler`'s `_requeue_lost`); a run this worker claimed and never
started has its start given back.

## Structs

### `Worker`

```mojo
struct Worker
```

Implements: `Movable`

Fields:

- `api: StateApi`
- `token: String`
- `pool: String`
- `name: String`
- `image: String`
- `max_runs: Int`
- `poll: Float64`
- `dir: String`
- `running: Dict[Int, _Running]`: The runs' processes, by pid.
- `keeping: Dict[Int, String]`: `keep-log`s going, by pid, with the stage they're keeping.
- `meters: Dict[Int, Meter]`: The runs' processes' samples, by pid, sent when each ends (`orca.usage`).
- `psi: Source`: Where this host's memory pressure is read from (`orca.pressure`).
- `admission: Admission`: Whether this worker claims runs for its host's memory pressure.
- `cgroups: Cgroups`: Where the runs' cgroups go (`orca.cgroup`), or why they don't.
- `contained: Dict[Int, Contained]`: The runs' processes' cgroups, by pid.
- `held: HeldWatch`: Since when they have been held at their memory limit.
- `stalled: Dict[Int, Bool]`: Those killed for being held there: out of memory at the reap.
- `unswept: List[String]`: Cgroups to remove once empty.
- `room_mb: Int`: The memory this host can grant, the limit of a run without a grant.

#### `__init__`

```mojo
def __init__(out self, url: String, token: String, pool: String, name: String, image: String, max_runs: Int, poll: Float64, dir: String)
```

#### `serve`

```mojo
def serve(mut self) -> Int
```

## Functions

### `worker_command`

```mojo
def worker_command(args: List[String], target: String) -> Int
```

`orca worker ...`, to the UI `target` unless `--remote` follows.

## Aliases

### `WORKER_USAGE`

```mojo
comptime WORKER_USAGE = "usage:\n  orca worker --pool <pool> [--remote <url>] [--token-file <path>]\n              [--max-runs N] [--poll-seconds S] [--dir <dir>]\n              [--image <image>] [--name <name>]\n              (the token: ORCA_WORKER_TOKEN, or --token-file)"
```