`orca.worker`

orca worker: runs a pool's runs on this machine, talking only to orca ui's API.

ORCA_WORKER_TOKEN=orca_... orca worker --pool bgent --remote https://orca.mlake.dev

Every --poll-seconds (2) the worker posts its heartbeat to /api/worker/claim (orca.workers), its name, image, CPUs, memory and the runs it has, and, while it has fewer than --max-runs (1), asks for one. A run it's given starts as the local runner starts one: the pipeline's binary (the path it was registered with, which the worker's image has too) with resume <run>, in a process group of its own, its output staged under --dir ($TMPDIR/orca-worker), with the pipeline's env, its secrets read here with ORCA_SECRET_FETCH and named in ORCA_SECRETS, and ORCA_STATE_URL and ORCA_STATE_TOKEN in place of a database: the run records itself over the API (orca.http_store). Its data and logs go where the worker's own ORCA_WAREHOUSE and ORCA_LOGS say, as any run's do. When the process ends, its output is appended to the run's driver.log by <binary> keep-log, which also fails the attempts a crash left running; a process that ended before claiming its run has the run failed, saying why (/api/worker/exited). Beside each run's process it starts <binary> sample (orca.sample, unless ORCA_SAMPLE_SECONDS=0), which writes this machine's metrics to <ORCA_LOGS>/<pipeline>/<run>/metrics/worker-<name>.jsonl, so the run's page has its Machines card, as a local run's does. Each poll the worker samples its runs' process trees, as the scheduler samples a local run's, and sends the samples with /api/worker/exited, so their attempts get their peak memory and CPU (orca.usage). Where the worker can make cgroups (orca.cgroup), each run gets one at its grant, and an out-of-memory kill is reported with /api/worker/exited, which fails the attempts saying so and raises what the pipeline asks for.

No database. The worker refuses to start with ORCA_DATABASE_URL set: nothing on a worker holds database credentials. Its token is ORCA_WORKER_TOKEN, or the first line of --token-file; one that's revoked, expired or for another pool than --pool is refused, and the worker exits (code 3).

Versions. The worker's image is --image, else ORCA_IMAGE. A run it's given whose pipeline's image isn't that is put back (/api/worker/release) rather than started, though the UI shouldn't hand it one (orca.workers.version_refusal).

Dying. A worker that's killed takes its runs with it: each run's heartbeat thread exits its process once its parent isn't the worker (ORCA_WORKER_PID), so the run's heartbeat goes stale and the scheduler queues it again, for this worker or another, whatever its trigger (orca.scheduler's _requeue_lost); a run this worker claimed and never started has its start given back.

Structs

`Worker`

struct Worker

Implements Movable

Fields: api: StateApi; token: String; pool: String; name: String; image: String; max_runs: Int; poll: Float64; dir: String; running: Dict[Int, _Running]: The runs' processes, by pid.; keeping: Dict[Int, String]: keep-logs going, by pid, with the stage they're keeping.; meters: Dict[Int, Meter]: The runs' processes' samples, by pid, sent when each ends (orca.usage).; psi: Source: Where this host's memory pressure is read from (orca.pressure).; admission: Admission: Whether this worker claims runs for its host's memory pressure.; cgroups: Cgroups: Where the runs' cgroups go (orca.cgroup), or why they don't.; contained: Dict[Int, Contained]: The runs' processes' cgroups, by pid.; held: HeldWatch: Since when they have been held at their memory limit.; stalled: Dict[Int, Bool]: Those killed for being held there: out of memory at the reap.; unswept: List[String]: Cgroups to remove once empty.; room_mb: Int: The memory this host can grant, the limit of a run without a grant.

`init`

def __init__(out self, url: String, token: String, pool: String, name: String, image: String, max_runs: Int, poll: Float64, dir: String)

`serve`

def serve(mut self) -> Int

Functions

`worker_command`

def worker_command(args: List[String], target: String) -> Int

orca worker ..., to the UI target unless --remote follows.

Aliases

comptime WORKER_USAGE = "usage:\n  orca worker --pool <pool> [--remote <url>] [--token-file <path>]\n              [--max-runs N] [--poll-seconds S] [--dir <dir>]\n              [--image <image>] [--name <name>]\n              (the token: ORCA_WORKER_TOKEN, or --token-file)"