# orca.control

> A module of morca's API.

Placement: where a pipeline's runs run (`orca register --placement`), and the control runner, which runs them on the scheduler's own host.

A pipeline's placement is `default` (stored as null: the scheduler's runner,
`ORCA_RUNNER`, as before), `ec2` (instances of their own, `orca.ec2`; on a
scheduler whose runner isn't `ec2` such runs wait, saying why),
`worker:<pool>` (a remote worker of that pool starts them, `orca.workers`,
never the scheduler) or `control`:
the scheduler starts the run as a child process of its own, the pipeline's
binary with `resume <run>` as the local runner does, which on AWS is in the
scheduler's container on the control instance (the binary is the project's
image's). Small runs then start in a second, without an instance to boot.

**Limits**, so a run can't starve Postgres, the scheduler or the UI:

- **Memory.** Each tick the scheduler adds up the resident memory of each
  control-placed run's process tree (`orca.sample.tree_rss`) and kills one
  over `ORCA_CONTROL_MEMORY_MB` (1024), failing its attempts with why. Its
  processes are also the first the kernel kills if memory runs out before
  that (`oom_score_adj` 1000, on Linux).
- **CPU.** Its processes run at `nice` `ORCA_CONTROL_NICE` (10), so the
  control instance's own processes win when the cores are busy.
- **How many.** At most `ORCA_CONTROL_MAX_RUNS` (2) control-placed runs, of
  all pipelines, run at once.
- **Pressure.** None starts while the control instance's newest minute
  (`orca.system`, from its sampler) has memory or its fullest disk over 85%.

A run held by either of the last two stays queued with the reason in
`runs.held`, which its page and JSON show. dista inside such a run is in
local mode, `ORCA_CONTROL_WORKERS` (1) forked workers.

**Secrets** (`--secret NAME=ID`) are read by the scheduler when it starts
the run, with `ORCA_SECRET_FETCH` as the instances read them, else from
Secrets Manager's API with the scheduler's credentials (the image has no
`aws` command), and put in the run's environment only, with `ORCA_SECRETS`
naming them so its logs mask them. Its driver's output is kept in a staging
file and appended to the run's `driver.log`, masked, by `<binary> keep-log`
once the process has ended, as on an EC2 head.

## Structs

### `ControlLimits`

```mojo
struct ControlLimits
```

What a control-placed run may use, from the scheduler's environment.

Implements: `Copyable`, `Movable`

Fields:

- `max_runs: Int`: `ORCA_CONTROL_MAX_RUNS` (2): control-placed runs at once.
- `memory_mb: Int`: `ORCA_CONTROL_MEMORY_MB` (1024): one run's resident memory.
- `workers: Int`: `ORCA_CONTROL_WORKERS` (1): its dista workers, forked locally.
- `nice: Int`: `ORCA_CONTROL_NICE` (10): its CPU priority, lowered by this.

#### `from_env`

```mojo
def from_env() -> Self
```

## Functions

### `pool_problem`

```mojo
def pool_problem(pool: String) -> String
```

Why `pool` can't name a pool of workers, or empty: 1 to 32 of `a-z`, `0-9`, `-` and `_`.

### `pool_of`

```mojo
def pool_of(placement: String) -> String
```

The pool a `worker:<pool>` placement names, or empty for any other.

### `parse_placement`

```mojo
def parse_placement(v: String) -> String
```

`--placement`'s value: `control`, `ec2`, `worker:<pool>` or `default`.

### `pressure_reason`

```mojo
def pressure_reason(now: Optional[Now]) -> String
```

Why no control-placed run may start now, given the control instance's newest minute, or empty if one may. Samples that are old or missing hold nothing: a scheduler without a sampler isn't stopped.

### `slots_reason`

```mojo
def slots_reason(running: Int, max_runs: Int) -> String
```

Why a control-placed run waits for one of the others to end.

### `control_stage`

```mojo
def control_stage(pipeline: String, run: Int, start: Int) -> String
```

Where a control-placed run's start, or a local run's with secrets, writes its output before `keep-log` appends it to the run's `driver.log`: under `ORCA_LOG_STAGE` (else `$TMPDIR/orca-logs`), never in the logs themselves, since it isn't masked yet.

### `run_env`

```mojo
def run_env(limits: ControlLimits, image: String, secrets: List[Tuple[String, String]]) -> List[String]
```

What a control-placed run's process gets beside the scheduler's environment: its local workers, its image (which its version records) and its secrets' values, `secrets` being (NAME, value) pairs, named in `ORCA_SECRETS`.

### `secrets_env`

```mojo
def secrets_env(secrets: List[Tuple[String, String]]) -> List[String]
```

A run's secrets as environment entries, `secrets` being (NAME, value) pairs: each `NAME=value`, then `ORCA_SECRETS` naming them, which `orca.logs.mask` masks the values by. Nothing for no secrets.

### `fetch_secret`

```mojo
def fetch_secret(id: String) -> String
```

The value of the secret `id`: what `ORCA_SECRET_FETCH <id>` prints, as the instances' scripts read it (trailing newlines dropped, as `$(...)` drops them), else Secrets Manager's `GetSecretValue`.

### `get_secret_value`

```mojo
def get_secret_value(id: String) -> String
```

Secrets Manager's `GetSecretValue` of `id` (`orca.awsjson`), with the credentials of the `AWS_*` environment or else the instance role; `ORCA_SECRETS_ENDPOINT` points it elsewhere, as the tests do at their stand-in (`tests/fake_ec2.py`).

## Aliases

### `CONTROL`

```mojo
comptime CONTROL = "control"
```

The placement, and the instance name of a control-placed run's samples.

### `HOLD_PERCENT`

```mojo
comptime HOLD_PERCENT = SIMD(85)
```

The control instance's memory or fullest disk, in percent, past which no control-placed run starts.

### `WORKER_PREFIX`

```mojo
comptime WORKER_PREFIX = "worker:"
```

A placement on a pool of remote workers (`orca.workers`): this and the pool's name.

### `MAX_POOL`

```mojo
comptime MAX_POOL = 32
```