# orca.workers

> A module of morca's API.

Remote workers, the server's side: worker tokens, claims, and the store calls a worker's runs make over `orca ui` (`orca.http_store`).

```
orca worker-token create --pool bgent [--days N] [--label L]   prints the token once
orca worker-token list                                          id, pool, label, worker, last seen
orca worker-token revoke <id>
```

A pipeline placed `worker:<pool>` (`placement` in orca.toml, or `orca
register --placement worker:<pool>`) has its runs started by a worker of
that pool (`orca worker`, `orca.worker`), never by the scheduler, which
keeps them queued, `held` "waiting for a worker of pool <pool>", requeues
one whose heartbeat went stale (a worker that died, or a laptop that went
to sleep) as it would start any dead run again, and fails one that has no
retries left. A run whose worker is gone (not seen for half the stale
limit, or seen since without it) is queued again whatever its trigger. A
pipeline with `fills = "all"` has its fill runs queued the same way, held
for a worker, at most `fill_max_runs` of them queued or running across the
pool (`orca.fill`).

**Tokens.** A worker token is an API token (`orca.tokens`: `orca_` and 64
hex digits, kept only as its SHA-256), in `orca.workers` rather than
`orca.api_tokens`, scoped to one pool: it can claim that pool's runs and
make the store calls of the runs it claimed, and nothing else; a user's
token gets nothing here, and a worker token gets nothing elsewhere. It
lasts `--days` (default and at most `ORCA_WORKER_TOKEN_DAYS`, 365). A
revoked token stays listed, as revoked, and is answered 401.

**The API** (`orca.api.worker`): `/api/worker/claim`, the worker's
heartbeat and its claims; `/api/worker/release`, a run it won't start;
`/api/worker/exited`, a run's process ended; `/api/worker/store/<call>`,
`store_call`, a `RunStore` call of a run it claimed.

A claim takes, under its pipeline's lock, a queued run whose pipeline isn't
paused and has fewer than `max_runs` live runs (a fill run: fewer than
`fill_max_runs` running fill runs, inside its `fill_window`, with fills on),
its other runs before its fill runs (the oldest of each pipeline's, so one
pipeline's queue at its cap never holds up another's), and starts it as the
scheduler starts a run: `running`, owner `worker`, no heartbeat yet (the
run's process claims it, within the start grace), one more start. Each
claim is a row of `orca.audit` (user `worker:<label>`, source `worker`).
**Versions.** A run needs its pipeline's image (`--image`, `image` in
orca.toml) to be the worker's (`ORCA_IMAGE`), both empty included: a worker
of another image isn't given it, and it stays queued, held saying why. The
worker checks the spec again before it starts anything (`orca.worker`).

## Structs

### `WorkerAuth`

```mojo
struct WorkerAuth
```

The worker a request's token is.

Implements: `Copyable`, `Movable`

Fields:

- `id: Int`
- `pool: String`
- `label: String`

#### `placement`

```mojo
def placement(self) -> String
```

#### `user`

```mojo
def user(self) -> String
```

Who it is in `orca.audit`.

### `WorkerRow`

```mojo
struct WorkerRow
```

A worker token, and what its worker last said of itself.

Implements: `Copyable`, `Movable`

Fields:

- `id: Int`
- `pool: String`
- `label: String`
- `state: String`: `live`, `revoked` or `expired`.
- `name: String`: The worker's host name, once it has claimed; else empty.
- `image: String`
- `last_seen: String`
- `seen_ago: Int`: Seconds since it was last seen; -1 if never.
- `cpus: Int`
- `memory_mb: Int`
- `running: String`: The runs it's running, comma-separated, as it said.
- `expires: String`
- `pressure: Optional[PressureRow]`: Its host's newest memory pressure reading (`orca.admission`), if it sent one.

### `WorkerRun`

```mojo
struct WorkerRun
```

What a worker is given to run: a start of a run, and how.

Implements: `Copyable`, `Movable`

Fields:

- `run: Int`
- `start: Int`
- `pipeline: String`
- `binary: String`
- `registry_hash: String`
- `partitions: String`
- `allow_new_code: Bool`
- `image: String`: The image it needs the worker to be (`version_refusal`).
- `env: List[Tuple[String, String]]`: The pipeline's own variables (`env` in orca.toml).
- `secrets: List[Tuple[String, String]]`: Its secrets: the variable and the ID the worker reads it by, with its own `ORCA_SECRET_FETCH`.
- `grant: Grant`: What this start was granted of the worker's host (`orca.resources`); none for a pipeline without a request, or a worker that didn't say what it can grant.

## Functions

### `version_refusal`

```mojo
def version_refusal(needed: String, have: String) -> String
```

Why a worker of image `have` can't run a run whose pipeline is image `needed`, or empty when they're the same.

### `worker_token_days`

```mojo
def worker_token_days(asked: Int = Int(0)) -> Int
```

`asked` if from 1 to `ORCA_WORKER_TOKEN_DAYS` (default 365, at most 3650), else that most.

### `create_worker_token`

```mojo
def create_worker_token(mut db: State, pool: String, label: String, days: Int, by: String) -> Issued
```

A new token for a worker of `pool`, which only this answer holds.

### `revoke_worker_token`

```mojo
def revoke_worker_token(mut db: State, id: Int) -> Bool
```

Revokes worker token `id`; False if it's no live one.

### `worker_of`

```mojo
def worker_of(mut db: State, token: String) -> Optional[WorkerAuth]
```

The live worker token `token` is, marked seen; None for any other.

### `worker_rows`

```mojo
def worker_rows(mut db: State) -> List[WorkerRow]
```

Every worker token, newest first.

### `ran_on`

```mojo
def ran_on(mut db: State, run: Int) -> String
```

The worker a run's latest start was claimed by, as `<name> (token <id>, <label>)`, or empty for a run no worker claimed.

### `worker_run_json`

```mojo
def worker_run_json(w: WorkerRun) -> String
```

### `worker_run_of`

```mojo
def worker_run_of(v: Value) -> WorkerRun
```

### `text_field`

```mojo
def text_field(v: Value, key: String) -> String
```

A field of a worker's JSON as text: a number's digits, empty for none or null.

### `claim`

```mojo
def claim(mut db: State, w: WorkerAuth, v: Value) -> String
```

`/api/worker/claim`'s answer: `{"run": <spec>|null, "refused": ...}`.

### `release`

```mojo
def release(mut db: State, w: WorkerAuth, v: Value) -> String
```

`/api/worker/release`: puts back a run the worker was given and won't start (its spec isn't for this worker's image), held saying why.

### `exited`

```mojo
def exited(mut db: State, w: WorkerAuth, v: Value) -> String
```

`/api/worker/exited`: fails a run whose process ended before it claimed it, as the scheduler's reap does for a local run, and records what the process's attempts used from the worker's samples.

### `store_call`

```mojo
def store_call(mut db: State, url: String, w: WorkerAuth, call: String, v: Value) -> String
```

A worker's run's `RunStore` call `call`, with the fields `v`, made on `PgStore`; the answer's JSON. Raises what the call raised.

### `workers_json`

```mojo
def workers_json(rows: List[WorkerRow]) -> String
```

## Aliases

### `WORKER_OWNER`

```mojo
comptime WORKER_OWNER = "worker"
```

A run's owner once a worker claimed it, until its process does.

### `DEFAULT_WORKER_TOKEN_DAYS`

```mojo
comptime DEFAULT_WORKER_TOKEN_DAYS = 365
```