# orca.pressure

> A module of morca's API.

Memory pressure (#202): Linux's pressure stall information (PSI) for memory, read each tick, and the admission rule built on it.

some avg10=1.53 avg60=0.87 avg300=0.24 total=1234567
    full avg10=0.00 avg60=0.12 avg300=0.03 total=34567

`some` is the share of the last 10, 60 and 300 seconds in which at least
one task was stalled waiting for memory (reclaim, swap-in, refaults);
`full` the share in which every runnable task was. Meta's TMO acts on the
same signal before memory runs out (Weiner et al., ASPLOS 2022).

**Which file** (`find_source`, once per process):

1. `ORCA_MEMORY_PRESSURE_FILE`, if set: that file, whatever it is (tests
   point it at a fixture);
2. on Linux, the cgroup v2 `memory.pressure` of the nearest cgroup, from
   this process's own up, whose `memory.max` is a limit: in a Docker
   container run with `--memory` that's the container's own
   (`/sys/fs/cgroup/memory.pressure`), which stalls when the container
   reaches its limit even with the host at ease;
3. the host's `/proc/pressure/memory`. Inside a container it is the host's
   (PSI isn't namespaced), which is what a container without a limit
   shares;
4. the own cgroup's `memory.pressure` though it has no limit, for a kernel
   with cgroup PSI but no readable `/proc/pressure`;
5. none: macOS, or a kernel built or booted without PSI (`psi=0`). Then
   nothing is held, and the scheduler says so once.

**Admission** (`Admission`): while `some avg10`, less what runs held at
their own memory limits explain (the most of their cgroups' `some avg10`),
is over the limit
(`ORCA_ADMIT_MEMORY_PRESSURE`, percent, default 10; 0 turns it off), no new
run starts on this host; runs already going go on. The hold ends once
`avg10` is under half the limit, or has stayed at most the limit for
`RESUME_SECONDS`, so a reading that hovers at the limit doesn't flap. A
hold that has lasted `ORCA_ADMIT_PRESSURE_GRACE_SECONDS` (600) with none of
orca's runs going on this host lets one run start at a time (`escape`): the
pressure isn't orca's, and orca shouldn't wait for it forever. The caller
logs and alerts once when that happens (`stuck`).

This module reads files and decides; it keeps nothing in a database, so a
step (`memory_pressure`, from `ORCA_MEMORY_PRESSURE_FILE`, which every run
on the local runner, the control host and a worker is given) and the
per-run cgroups of #201 can use it alone.

## Structs

### `Stall`

```mojo
struct Stall
```

One line of a PSI file: the percent of time stalled, averaged over 10, 60 and 300 seconds, and the total stalled, in microseconds.

Implements: `Copyable`, `ImplicitlyCopyable`, `Movable`, `Writable`

Fields:

- `avg10: Float64`
- `avg60: Float64`
- `avg300: Float64`
- `total: Int`

#### `zero`

```mojo
def zero() -> Self
```

### `Psi`

```mojo
struct Psi
```

A PSI file's `some` and `full` lines; `has_full` is False for a file without one (`/proc/pressure/cpu` before 5.13, or a fixture).

Implements: `Copyable`, `ImplicitlyCopyable`, `Movable`, `Writable`

Fields:

- `some: Stall`
- `full: Stall`
- `has_full: Bool`

### `Source`

```mojo
struct Source
```

The PSI file a host's memory pressure is read from: `kind` is `file` (`ORCA_MEMORY_PRESSURE_FILE`), `cgroup` (a cgroup with a memory limit), `host` (`/proc/pressure/memory`), `cgroup-unlimited`, or empty with no `path` for none.

Implements: `Boolable`, `Copyable`, `Movable`, `Writable`

Fields:

- `path: String`
- `kind: String`
- `limit_bytes: Int`: The cgroup's `memory.max` for `cgroup`; else 0.

#### `__bool__`

```mojo
def __bool__(self) -> Bool
```

#### `none`

```mojo
def none() -> Self
```

#### `describe`

```mojo
def describe(self) -> String
```

In words, for a log line and Ops › Control.

### `Admission`

```mojo
struct Admission
```

Whether this host starts new runs, from each reading of its memory pressure (`observe`), with hysteresis. Times are seconds on any clock that only goes forward.

Implements: `Copyable`, `Movable`

Fields:

- `limit: Float64`: `some avg10` percent over which runs wait; 0 for off.
- `grace: Float64`: How long a hold lasts, with none of orca's runs going, before one may start anyway.
- `holding: Bool`
- `held_since: Float64`
- `calm_since: Float64`: While holding, when `avg10` came to at most the limit; -1 if it isn't.
- `reading: Optional[Psi]`: The newest reading; None without one.
- `hold: String`: While holding: the reason a run waits (`WAITING_PRESSURE`...); else empty.
- `escape: Bool`: Holding past `grace` with nothing of orca's going: one run may start while none is.
- `stuck: Bool`: `escape` became true at this reading: log it and alert, once a hold.
- `released: Bool`: A hold ended at this reading.

#### `__init__`

```mojo
def __init__(out self, limit: Float64, grace: Float64 = 600)
```

#### `from_env`

```mojo
def from_env() -> Self
```

#### `observe`

```mojo
def observe(mut self, psi: Optional[Psi], now: Float64, idle: Bool, own: Float64 = 0)
```

Takes a reading (None: none could be read, which holds nothing), at `now`, with `idle` whether none of orca's runs is going on this host. `own` is the part of `some avg10` that runs held at their own memory limits explain (`orca.cgroup.own_limit_pressure`), which holds nothing: the host may have memory to spare, and such a run is killed and retried bigger in its own time.

#### `held`

```mojo
def held(self) -> Bool
```

#### `allows`

```mojo
def allows(self, idle: Bool) -> Int
```

How many starts this host may make now: none while holding, but one while `escape` and `idle`; -1 for no limit from pressure.

## Functions

### `parse_psi`

```mojo
def parse_psi(text: String) -> Psi
```

A PSI file's text; raises without a `some` line, or on a line that isn't `some|full avg10=.. avg60=.. avg300=.. total=..`.

### `read_psi`

```mojo
def read_psi(path: String) -> Optional[Psi]
```

The PSI file at `path`, or None if it can't be read or parsed (no file, PSI off in the kernel, a fixture half written).

### `memory_pressure`

```mojo
def memory_pressure() -> Optional[Psi]
```

This run's host's memory pressure, from the file `ORCA_MEMORY_PRESSURE_FILE` names (which orca gives every run it starts where there is one), or None. A step can poll it and lower its own parallelism while `some.avg10` is high.

### `own_cgroup`

```mojo
def own_cgroup(proc_self_cgroup: String) -> String
```

The cgroup v2 path in `/proc/self/cgroup`'s text (`0::/path`), or empty for none (cgroup v1 only).

### `pick_source`

```mojo
def pick_source(override: String, cgroup_dirs: List[String], limits: List[Int], has_pressure: List[Bool], host_ok: Bool) -> Source
```

The choice `find_source` makes, from what it found: `override` (`ORCA_MEMORY_PRESSURE_FILE`), the cgroup directories from this process's own up to the root of the mount, each one's `memory.max` in bytes (0 for none) and whether it has a readable `memory.pressure`, and whether `/proc/pressure/memory` reads.

### `cgroup_dirs`

```mojo
def cgroup_dirs(rel: String, root: String = CGROUP_ROOT) -> List[String]
```

The directories under `root` of cgroup `rel` and its parents, own first, to `root` itself. With a private cgroup namespace (Docker's default on cgroup v2) `rel` is `/` and `root` is the container's cgroup.

### `find_source`

```mojo
def find_source() -> Source
```

Where this host's memory pressure is read from; see the module's docs. Reads files, so it's called once.

### `pct_text`

```mojo
def pct_text(x: Float64) -> String
```

A percent for a reason: whole from 10, else to a tenth without a trailing `.0`: `23`, `7.5`, `0`.

### `admit_limit`

```mojo
def admit_limit() -> Float64
```

`ORCA_ADMIT_MEMORY_PRESSURE`, percent; 10 when unset, and 0 (off) for a value that isn't a number from 0.

### `grace_seconds`

```mojo
def grace_seconds() -> Float64
```

### `pressure_env`

```mojo
def pressure_env(s: Source) -> List[String]
```

`ORCA_MEMORY_PRESSURE_FILE=<path>` for a run's process, so a step can poll its host's pressure (`memory_pressure`); nothing without one.

## Aliases

### `PRESSURE_FILE`

```mojo
comptime PRESSURE_FILE = "ORCA_MEMORY_PRESSURE_FILE"
```

The file a run polls for its host's memory pressure, and the scheduler's override of which file it reads.

### `ADMIT_LIMIT`

```mojo
comptime ADMIT_LIMIT = "ORCA_ADMIT_MEMORY_PRESSURE"
```

The `some avg10` percent over which no new run starts; 0 turns it off.

### `DEFAULT_LIMIT`

```mojo
comptime DEFAULT_LIMIT = SIMD(10)
```

### `GRACE`

```mojo
comptime GRACE = "ORCA_ADMIT_PRESSURE_GRACE_SECONDS"
```

### `DEFAULT_GRACE`

```mojo
comptime DEFAULT_GRACE = SIMD(600)
```

### `RESUME_SECONDS`

```mojo
comptime RESUME_SECONDS = SIMD(30)
```

How long `avg10` stays at most the limit before a hold ends anyway.

### `WAITING_PRESSURE`

```mojo
comptime WAITING_PRESSURE = "waiting: memory pressure "
```

How a run held by memory pressure's reason starts.

### `HOST_PSI`

```mojo
comptime HOST_PSI = "/proc/pressure/memory"
```

### `CGROUP_ROOT`

```mojo
comptime CGROUP_ROOT = "/sys/fs/cgroup"
```