morca

morca · API

orca.control

Placement: where a pipeline's runs run (orca register --placement), and the control runner, which runs them on the scheduler's own host.

A pipeline's placement is default (stored as null: the scheduler's runner, ORCA_RUNNER, as before), ec2 (instances of their own, orca.ec2; on a scheduler whose runner isn't ec2 such runs wait, saying why), worker:<pool> (a remote worker of that pool starts them, orca.workers, never the scheduler) or control: the scheduler starts the run as a child process of its own, the pipeline's binary with resume <run> as the local runner does, which on AWS is in the scheduler's container on the control instance (the binary is the project's image's). Small runs then start in a second, without an instance to boot.

Limits, so a run can't starve Postgres, the scheduler or the UI:

  • Memory. Each tick the scheduler adds up the resident memory of each control-placed run's process tree (orca.sample.tree_rss) and kills one over ORCA_CONTROL_MEMORY_MB (1024), failing its attempts with why. Its processes are also the first the kernel kills if memory runs out before that (oom_score_adj 1000, on Linux).
  • CPU. Its processes run at nice ORCA_CONTROL_NICE (10), so the control instance's own processes win when the cores are busy.
  • How many. At most ORCA_CONTROL_MAX_RUNS (2) control-placed runs, of all pipelines, run at once.
  • Pressure. None starts while the control instance's newest minute (orca.system, from its sampler) has memory or its fullest disk over 85%.

A run held by either of the last two stays queued with the reason in runs.held, which its page and JSON show. dista inside such a run is in local mode, ORCA_CONTROL_WORKERS (1) forked workers.

Secrets (--secret NAME=ID) are read by the scheduler when it starts the run, with ORCA_SECRET_FETCH as the instances read them, else from Secrets Manager's API with the scheduler's credentials (the image has no aws command), and put in the run's environment only, with ORCA_SECRETS naming them so its logs mask them. Its driver's output is kept in a staging file and appended to the run's driver.log, masked, by <binary> keep-log once the process has ended, as on an EC2 head.

Structs

ControlLimits

struct ControlLimits

What a control-placed run may use, from the scheduler's environment.

Implements Copyable, Movable

Fields
max_runs: Int: ORCA_CONTROL_MAX_RUNS (2): control-placed runs at once.
memory_mb: Int: ORCA_CONTROL_MEMORY_MB (1024): one run's resident memory.
workers: Int: ORCA_CONTROL_WORKERS (1): its dista workers, forked locally.
nice: Int: ORCA_CONTROL_NICE (10): its CPU priority, lowered by this.

from_env

def from_env() -> Self

Functions

pool_problem

def pool_problem(pool: String) -> String

Why pool can't name a pool of workers, or empty: 1 to 32 of a-z, 0-9, - and _.

pool_of

def pool_of(placement: String) -> String

The pool a worker:<pool> placement names, or empty for any other.

parse_placement

def parse_placement(v: String) -> String

--placement's value: control, ec2, worker:<pool> or default.

pressure_reason

def pressure_reason(now: Optional[Now]) -> String

Why no control-placed run may start now, given the control instance's newest minute, or empty if one may. Samples that are old or missing hold nothing: a scheduler without a sampler isn't stopped.

slots_reason

def slots_reason(running: Int, max_runs: Int) -> String

Why a control-placed run waits for one of the others to end.

control_stage

def control_stage(pipeline: String, run: Int, start: Int) -> String

Where a control-placed run's start, or a local run's with secrets, writes its output before keep-log appends it to the run's driver.log: under ORCA_LOG_STAGE (else $TMPDIR/orca-logs), never in the logs themselves, since it isn't masked yet.

run_env

def run_env(limits: ControlLimits, image: String, secrets: List[Tuple[String, String]]) -> List[String]

What a control-placed run's process gets beside the scheduler's environment: its local workers, its image (which its version records) and its secrets' values, secrets being (NAME, value) pairs, named in ORCA_SECRETS.

secrets_env

def secrets_env(secrets: List[Tuple[String, String]]) -> List[String]

A run's secrets as environment entries, secrets being (NAME, value) pairs: each NAME=value, then ORCA_SECRETS naming them, which orca.logs.mask masks the values by. Nothing for no secrets.

fetch_secret

def fetch_secret(id: String) -> String

The value of the secret id: what ORCA_SECRET_FETCH <id> prints, as the instances' scripts read it (trailing newlines dropped, as $(...) drops them), else Secrets Manager's GetSecretValue.

get_secret_value

def get_secret_value(id: String) -> String

Secrets Manager's GetSecretValue of id (orca.awsjson), with the credentials of the AWS_* environment or else the instance role; ORCA_SECRETS_ENDPOINT points it elsewhere, as the tests do at their stand-in (tests/fake_ec2.py).

Aliases

comptime CONTROL = "control"

The placement, and the instance name of a control-placed run's samples.

comptime HOLD_PERCENT = SIMD(85)

The control instance's memory or fullest disk, in percent, past which no control-placed run starts.

comptime WORKER_PREFIX = "worker:"

A placement on a pool of remote workers (orca.workers): this and the pool's name.

comptime MAX_POOL = 32