# orca.cgroup > A module of morca's API. Containment (#201): a cgroup v2 memory limit per run, and retrying bigger after an out-of-memory kill. A local or control-placed run (and a remote worker's, `orca.worker`) gets a cgroup of its own, `orca-run---`, beside the leaf `orca` the scheduler moved itself into. Its process is placed in it between `fork` and `exec` (`orca.proc.spawn`: the parent opens the cgroup's `cgroup.procs` before forking, and the child only `write`s `0` to it, which is async-signal-safe; the descriptor closes at `exec`), so everything the run starts is in it from its first instruction. - `memory.high` is the run's **limit**: its grant (`orca.resources`), else its request's maximum, else `ORCA_RUN_MEMORY_MB` (default: what this host can grant, its capacity less the reserve); on the control placement at most `ORCA_CONTROL_MEMORY_MB`. Past it the kernel throttles the run and reclaims its memory before anything dies. - `memory.max` is the limit plus `ORCA_RUN_MEMORY_MARGIN` percent (25): past it the kernel kills the run's processes, all of them (`memory.oom.group`), and none of anything else's. `memory.swap.max` is 0, so the limit is one of memory, not of memory and swap. - **Held at the limit.** A run past `memory.high` whose memory the kernel can't reclaim (anonymous, and no swap) is throttled, not killed: each allocation waits, so it may creep towards `memory.max` for hours. The scheduler checks its runs each tick: one at its limit (`at_limit`) with every task stalled on memory most of the time (the cgroup's `memory.pressure` `full avg10` at least `HELD_PRESSURE`) for `ORCA_RUN_MEMORY_HELD_SECONDS` (120; 0: never) in a row is killed, and counts as out of memory as below. - At the reap, `memory.events`' `oom_kill` says whether the kernel killed them for it: the attempts fail with `out of memory at ` rather than signal 9. - **Retrying bigger.** Such a kill raises the pipeline's memory (`orca.pipelines.memory_raised_mb`) to the limit times `ORCA_RUN_MEMORY_GROWTH` (1.5), at most its request's maximum (with no request, what the host can grant; on the control placement, the control cap). The retry, and every later run of the pipeline, asks for at least that: it's its request's minimum, which admission waits for, or with no request its limit. Registering a different `memory_gb` forgets it. - **Stall time.** At the reap, the cgroup's `memory.pressure` totals say how long the run's tasks waited on memory over the start: some of them (`some`) and all of them at once (`full`). They go on its attempts (`memory_stall_ms`, `memory_stall_full_ms`; for #281): how tight a grant was, where a kill only says it was too tight. - The cgroup is removed (`rmdir`) once the tree has left it, then or on a later tick; a scheduler that starts removes the empty ones a previous scheduler left (`stale`). Where orca can't make cgroups (macOS, cgroup v1, a container whose `/sys/fs/cgroup` is read-only or not delegated, `ORCA_CGROUPS=0`), it says why once and runs as before: `oom_score_adj` 1000 and, for control-placed runs, the sampled `ORCA_CONTROL_MEMORY_MB` cap. docs/hetzner.md says what the Hetzner deploy needs. ## Structs ### `MemoryKnobs` ```mojo struct MemoryKnobs ``` The scheduler's (or worker's) settings for its runs' limits. Implements: `Copyable`, `ImplicitlyCopyable`, `Movable` Fields: - `default_mb: Int`: `ORCA_RUN_MEMORY_MB`: the limit of a run without a request; 0 for what the host can grant. - `margin_pct: Int`: `ORCA_RUN_MEMORY_MARGIN` (25): `memory.max` this many percent over the limit. - `growth_pct: Int`: `ORCA_RUN_MEMORY_GROWTH` (1.5) as a percentage: a retry's memory after an out-of-memory kill. - `held_s: Int`: `ORCA_RUN_MEMORY_HELD_SECONDS` (120): how long a run may be held at its limit (`held`) before it's killed as out of memory; 0: never. #### `from_env` ```mojo def from_env() -> Self ``` ### `HeldWatch` ```mojo struct HeldWatch ``` Since when each contained run has been held at its memory limit (`held`), to say which have been held `ORCA_RUN_MEMORY_HELD_SECONDS` in a row: the scheduler's and a worker's runs alike. Implements: `Copyable`, `Movable` Fields: - `since: Dict[Int, Int]`: A run's process to the first check (monotonic ns) of the unbroken stretch it has been held; -1 once `due` has named it. #### `__init__` ```mojo def __init__(out self) ``` #### `due` ```mojo def due(mut self, contained: Dict[Int, Contained], held_s: Int) -> List[Int] ``` The processes of `contained` held for `held_s` seconds in a row, each named once; none when `held_s` is 0. ### `Contained` ```mojo struct Contained ``` A run's cgroup, as its starter keeps it until the process is reaped. Implements: `Copyable`, `Movable` Fields: - `run: Int` - `start: Int` - `pipeline: String` - `path: String` - `limit_mb: Int` - `kill_mb: Int`: Its `memory.max` in MB. - `ceiling_mb: Int`: The most a retry may ask for (`ceiling_mb`). - `grant_mb: Int` ### `Cgroups` ```mojo struct Cgroups ``` Where this process makes its runs' cgroups, or why it can't. Implements: `Copyable`, `Movable` Fields: - `base: String`: The cgroup the runs' cgroups go in; empty when off. - `why: String`: Why it's off, or how it's on, for the log. - `knobs: MemoryKnobs` #### `__init__` ```mojo def __init__(out self, base: String, why: String, knobs: MemoryKnobs) ``` #### `off` ```mojo def off(why: String) -> Self ``` #### `on` ```mojo def on(self) -> Bool ``` #### `setup` ```mojo def setup(knobs: MemoryKnobs) -> Self ``` Finds or makes the cgroup runs' cgroups go in: this process's own, after moving what's in it into a leaf `orca` (or, if it's in such a leaf already, the one above), with the memory controller on for its children. Never raises: off says why. #### `make` ```mojo def make(self, run: Int, start: Int, limit_mb: Int) -> Tuple[String, Int] ``` Makes a start's cgroup with `memory.high` at `limit_mb` and `memory.max` the margin above, and returns its path and `memory.max` in MB. #### `oom_kills` ```mojo def oom_kills(path: String) -> Int ``` The processes the kernel killed in the cgroup for its memory. #### `full_pressure` ```mojo def full_pressure(path: String) -> Float64 ``` The cgroup's `full avg10` (`memory.pressure`); -1 if it can't be read (a kernel without PSI). #### `stall_ms` ```mojo def stall_ms(path: String) -> Tuple[Int, Int] ``` How long the cgroup's tasks waited on memory, in ms: some of them, and all of them at once; each -1 if it can't be read. #### `current` ```mojo def current(path: String) -> Int ``` The cgroup's memory now, in bytes (`memory.current`); -1 if it can't be read. #### `remove` ```mojo def remove(path: String) -> Bool ``` Removes the cgroup once nothing is in it; whether it's gone. #### `stale` ```mojo def stale(self) -> List[String] ``` The runs' cgroups no live process of the same pid made, or this one before it started again (a restart of the container keeps the pid), to be removed once empty. ## Functions ### `max_mb` ```mojo def max_mb(limit_mb: Int, margin_pct: Int) -> Int ``` `memory.max` for a limit of `limit_mb`: the margin above it, rounded up to a whole MB. ### `raised_mb` ```mojo def raised_mb(limit_mb: Int, growth_pct: Int, ceiling_mb: Int) -> Int ``` What a run killed at `limit_mb` asks for next: `growth_pct` percent of it, rounded up, at most `ceiling_mb` (when over 0), and never less than `limit_mb`. ### `start_limit_mb` ```mojo def start_limit_mb(grant_mb: Int, request_max_mb: Int, default_mb: Int, raised: Int, cap_mb: Int) -> Int ``` A start's limit: its grant, else its request's maximum, else the larger of `default_mb` and the pipeline's `raised` memory; at most `cap_mb` when that's over 0 (the control placement's cap). ### `ceiling_mb` ```mojo def ceiling_mb(request_max_mb: Int, room_mb: Int, cap_mb: Int) -> Int ``` The most a retry may ask for: the request's maximum, else what the host can grant; at most `cap_mb` when that's over 0. ### `raised_request` ```mojo def raised_request(min_mb: Int, max_mb: Int, raised: Int) -> Int ``` A request's minimum, `min_mb`, raised to the pipeline's `raised` memory, up to its maximum. ### `oom_text` ```mojo def oom_text(limit_mb: Int, kill_mb: Int, next_mb: Int, stalled: Bool = False) -> String ``` An attempt's error after the kernel killed its run for its memory, or, `stalled`, after the scheduler killed it while it was held at its limit (`held`, or a heartbeat that went stale there). ### `own_path` ```mojo def own_path(proc_self_cgroup: String) -> String ``` The cgroup v2 path in `/proc/self/cgroup`'s text (its `0::` line), or empty with no unified hierarchy (cgroup v1). ### `event` ```mojo def event(text: String, name: String) -> Int ``` The count `name` in a `memory.events` file's `key value` lines; 0 without it. ### `has_word` ```mojo def has_word(text: String, word: String) -> Bool ``` Whether `cgroup.controllers` or `cgroup.subtree_control` names `word`. ### `at_limit` ```mojo def at_limit(current_bytes: Int, limit_mb: Int) -> Bool ``` Whether a cgroup's memory, `current_bytes`, is at its limit (within 5%): one throttled there. ### `full_avg10` ```mojo def full_avg10(text: String) -> Float64 ``` `full avg10` from a `memory.pressure` file's `some ...` and `full ...` lines; -1 without it. ### `avg10_of` ```mojo def avg10_of(text: String, kind: String) -> Float64 ``` `avg10` of the `some` or `full` line (`kind`) of a `memory.pressure` file; -1 without it. ### `stall_total_us` ```mojo def stall_total_us(text: String, kind: String) -> Int ``` `total=` of the `some` or `full` line (`kind`) of a `memory.pressure` file: microseconds its tasks waited on memory; -1 without it. ### `held` ```mojo def held(current_bytes: Int, limit_mb: Int, full_pct: Float64) -> Bool ``` Whether a run's cgroup is held at its limit now: at it (`at_limit`) and its tasks all stalled on memory at least `HELD_PRESSURE` percent of the time. ### `own_limit_pressure` ```mojo def own_limit_pressure(contained: Dict[Int, Contained]) -> Float64 ``` The most `some avg10` of the contained runs now at their own memory limit (`at_limit`): the stall their limits cause, which the host's pressure counts too, though the host may have memory to spare. 0 for none. ### `run_name` ```mojo def run_name(run: Int, start: Int, owner: Int) -> String ``` A run's cgroup's name: its run, its start and the pid of the scheduler (or worker) that made it. ### `owner_of` ```mojo def owner_of(name: String) -> Int ``` The pid in a run cgroup's name, or -1 for another name. ### `put` ```mojo def put(path: String, text: String) -> Bool ``` Writes `text` to a cgroup file at once; whether the kernel took it (stdio, as `orca.proc`: `open` is declared elsewhere with another signature). ### `growth_pct_of` ```mojo def growth_pct_of(s: String) -> Int ``` `ORCA_RUN_MEMORY_GROWTH`'s factor, from 1 (`1.5`), as a percentage. ### `held_text` ```mojo def held_text(c: Contained, held_s: Int) -> String ``` Why a held run's process was killed, for the log. ## Aliases ### `ROOT` ```mojo comptime ROOT = "/sys/fs/cgroup" ``` ### `LEAF` ```mojo comptime LEAF = "orca" ``` The cgroup the scheduler moves itself (and what shared its cgroup) into, so the one above can hold the runs' cgroups (cgroup v2 lets a cgroup with controllers for its children hold no processes). ### `RUN_PREFIX` ```mojo comptime RUN_PREFIX = "orca-run-" ``` ### `HELD_PRESSURE` ```mojo comptime HELD_PRESSURE = SIMD(50) ``` The `full avg10` (percent of the last 10 s all the run's tasks were stalled on memory) at or above which a run at its limit is held there.