morca

morca · API

orca.cgroup

Containment (#201): a cgroup v2 memory limit per run, and retrying bigger after an out-of-memory kill.

A local or control-placed run (and a remote worker's, orca.worker) gets a cgroup of its own, orca-run-<run>-<start>-<owner pid>, beside the leaf orca the scheduler moved itself into. Its process is placed in it between fork and exec (orca.proc.spawn: the parent opens the cgroup's cgroup.procs before forking, and the child only writes 0 to it, which is async-signal-safe; the descriptor closes at exec), so everything the run starts is in it from its first instruction.

  • memory.high is the run's limit: its grant (orca.resources), else its request's maximum, else ORCA_RUN_MEMORY_MB (default: what this host can grant, its capacity less the reserve); on the control placement at most ORCA_CONTROL_MEMORY_MB. Past it the kernel throttles the run and reclaims its memory before anything dies.
  • memory.max is the limit plus ORCA_RUN_MEMORY_MARGIN percent (25): past it the kernel kills the run's processes, all of them (memory.oom.group), and none of anything else's. memory.swap.max is 0, so the limit is one of memory, not of memory and swap.
  • Held at the limit. A run past memory.high whose memory the kernel can't reclaim (anonymous, and no swap) is throttled, not killed: each allocation waits, so it may creep towards memory.max for hours. The scheduler checks its runs each tick: one at its limit (at_limit) with every task stalled on memory most of the time (the cgroup's memory.pressure full avg10 at least HELD_PRESSURE) for ORCA_RUN_MEMORY_HELD_SECONDS (120; 0: never) in a row is killed, and counts as out of memory as below.
  • At the reap, memory.events' oom_kill says whether the kernel killed them for it: the attempts fail with out of memory at <limit> rather than signal 9.
  • Retrying bigger. Such a kill raises the pipeline's memory (orca.pipelines.memory_raised_mb) to the limit times ORCA_RUN_MEMORY_GROWTH (1.5), at most its request's maximum (with no request, what the host can grant; on the control placement, the control cap). The retry, and every later run of the pipeline, asks for at least that: it's its request's minimum, which admission waits for, or with no request its limit. Registering a different memory_gb forgets it.
  • Stall time. At the reap, the cgroup's memory.pressure totals say how long the run's tasks waited on memory over the start: some of them (some) and all of them at once (full). They go on its attempts (memory_stall_ms, memory_stall_full_ms; for #281): how tight a grant was, where a kill only says it was too tight.
  • The cgroup is removed (rmdir) once the tree has left it, then or on a later tick; a scheduler that starts removes the empty ones a previous scheduler left (stale).

Where orca can't make cgroups (macOS, cgroup v1, a container whose /sys/fs/cgroup is read-only or not delegated, ORCA_CGROUPS=0), it says why once and runs as before: oom_score_adj 1000 and, for control-placed runs, the sampled ORCA_CONTROL_MEMORY_MB cap. docs/hetzner.md says what the Hetzner deploy needs.

Structs

MemoryKnobs

struct MemoryKnobs

The scheduler's (or worker's) settings for its runs' limits.

Implements Copyable, ImplicitlyCopyable, Movable

Fields
default_mb: Int: ORCA_RUN_MEMORY_MB: the limit of a run without a request; 0 for what the host can grant.
margin_pct: Int: ORCA_RUN_MEMORY_MARGIN (25): memory.max this many percent over the limit.
growth_pct: Int: ORCA_RUN_MEMORY_GROWTH (1.5) as a percentage: a retry's memory after an out-of-memory kill.
held_s: Int: ORCA_RUN_MEMORY_HELD_SECONDS (120): how long a run may be held at its limit (held) before it's killed as out of memory; 0: never.

from_env

def from_env() -> Self

HeldWatch

struct HeldWatch

Since when each contained run has been held at its memory limit (held), to say which have been held ORCA_RUN_MEMORY_HELD_SECONDS in a row: the scheduler's and a worker's runs alike.

Implements Copyable, Movable

Fields
since: Dict[Int, Int]: A run's process to the first check (monotonic ns) of the unbroken stretch it has been held; -1 once due has named it.

__init__

def __init__(out self)

due

def due(mut self, contained: Dict[Int, Contained], held_s: Int) -> List[Int]

The processes of contained held for held_s seconds in a row, each named once; none when held_s is 0.

Contained

struct Contained

A run's cgroup, as its starter keeps it until the process is reaped.

Implements Copyable, Movable

Fields
run: Int
start: Int
pipeline: String
path: String
limit_mb: Int
kill_mb: Int: Its memory.max in MB.
ceiling_mb: Int: The most a retry may ask for (ceiling_mb).
grant_mb: Int

Cgroups

struct Cgroups

Where this process makes its runs' cgroups, or why it can't.

Implements Copyable, Movable

Fields
base: String: The cgroup the runs' cgroups go in; empty when off.
why: String: Why it's off, or how it's on, for the log.
knobs: MemoryKnobs

__init__

def __init__(out self, base: String, why: String, knobs: MemoryKnobs)

off

def off(why: String) -> Self

on

def on(self) -> Bool

setup

def setup(knobs: MemoryKnobs) -> Self

Finds or makes the cgroup runs' cgroups go in: this process's own, after moving what's in it into a leaf orca (or, if it's in such a leaf already, the one above), with the memory controller on for its children. Never raises: off says why.

make

def make(self, run: Int, start: Int, limit_mb: Int) -> Tuple[String, Int]

Makes a start's cgroup with memory.high at limit_mb and memory.max the margin above, and returns its path and memory.max in MB.

oom_kills

def oom_kills(path: String) -> Int

The processes the kernel killed in the cgroup for its memory.

full_pressure

def full_pressure(path: String) -> Float64

The cgroup's full avg10 (memory.pressure); -1 if it can't be read (a kernel without PSI).

stall_ms

def stall_ms(path: String) -> Tuple[Int, Int]

How long the cgroup's tasks waited on memory, in ms: some of them, and all of them at once; each -1 if it can't be read.

current

def current(path: String) -> Int

The cgroup's memory now, in bytes (memory.current); -1 if it can't be read.

remove

def remove(path: String) -> Bool

Removes the cgroup once nothing is in it; whether it's gone.

stale

def stale(self) -> List[String]

The runs' cgroups no live process of the same pid made, or this one before it started again (a restart of the container keeps the pid), to be removed once empty.

Functions

max_mb

def max_mb(limit_mb: Int, margin_pct: Int) -> Int

memory.max for a limit of limit_mb: the margin above it, rounded up to a whole MB.

raised_mb

def raised_mb(limit_mb: Int, growth_pct: Int, ceiling_mb: Int) -> Int

What a run killed at limit_mb asks for next: growth_pct percent of it, rounded up, at most ceiling_mb (when over 0), and never less than limit_mb.

start_limit_mb

def start_limit_mb(grant_mb: Int, request_max_mb: Int, default_mb: Int, raised: Int, cap_mb: Int) -> Int

A start's limit: its grant, else its request's maximum, else the larger of default_mb and the pipeline's raised memory; at most cap_mb when that's over 0 (the control placement's cap).

ceiling_mb

def ceiling_mb(request_max_mb: Int, room_mb: Int, cap_mb: Int) -> Int

The most a retry may ask for: the request's maximum, else what the host can grant; at most cap_mb when that's over 0.

raised_request

def raised_request(min_mb: Int, max_mb: Int, raised: Int) -> Int

A request's minimum, min_mb, raised to the pipeline's raised memory, up to its maximum.

oom_text

def oom_text(limit_mb: Int, kill_mb: Int, next_mb: Int, stalled: Bool = False) -> String

An attempt's error after the kernel killed its run for its memory, or, stalled, after the scheduler killed it while it was held at its limit (held, or a heartbeat that went stale there).

own_path

def own_path(proc_self_cgroup: String) -> String

The cgroup v2 path in /proc/self/cgroup's text (its 0:: line), or empty with no unified hierarchy (cgroup v1).

event

def event(text: String, name: String) -> Int

The count name in a memory.events file's key value lines; 0 without it.

has_word

def has_word(text: String, word: String) -> Bool

Whether cgroup.controllers or cgroup.subtree_control names word.

at_limit

def at_limit(current_bytes: Int, limit_mb: Int) -> Bool

Whether a cgroup's memory, current_bytes, is at its limit (within 5%): one throttled there.

full_avg10

def full_avg10(text: String) -> Float64

full avg10 from a memory.pressure file's some ... and full ... lines; -1 without it.

avg10_of

def avg10_of(text: String, kind: String) -> Float64

avg10 of the some or full line (kind) of a memory.pressure file; -1 without it.

stall_total_us

def stall_total_us(text: String, kind: String) -> Int

total= of the some or full line (kind) of a memory.pressure file: microseconds its tasks waited on memory; -1 without it.

held

def held(current_bytes: Int, limit_mb: Int, full_pct: Float64) -> Bool

Whether a run's cgroup is held at its limit now: at it (at_limit) and its tasks all stalled on memory at least HELD_PRESSURE percent of the time.

own_limit_pressure

def own_limit_pressure(contained: Dict[Int, Contained]) -> Float64

The most some avg10 of the contained runs now at their own memory limit (at_limit): the stall their limits cause, which the host's pressure counts too, though the host may have memory to spare. 0 for none.

run_name

def run_name(run: Int, start: Int, owner: Int) -> String

A run's cgroup's name: its run, its start and the pid of the scheduler (or worker) that made it.

owner_of

def owner_of(name: String) -> Int

The pid in a run cgroup's name, or -1 for another name.

put

def put(path: String, text: String) -> Bool

Writes text to a cgroup file at once; whether the kernel took it (stdio, as orca.proc: open is declared elsewhere with another signature).

growth_pct_of

def growth_pct_of(s: String) -> Int

ORCA_RUN_MEMORY_GROWTH's factor, from 1 (1.5), as a percentage.

held_text

def held_text(c: Contained, held_s: Int) -> String

Why a held run's process was killed, for the log.

Aliases

comptime ROOT = "/sys/fs/cgroup"
comptime LEAF = "orca"

The cgroup the scheduler moves itself (and what shared its cgroup) into, so the one above can hold the runs' cgroups (cgroup v2 lets a cgroup with controllers for its children hold no processes).

comptime RUN_PREFIX = "orca-run-"
comptime HELD_PRESSURE = SIMD(50)

The full avg10 (percent of the last 10 s all the run's tasks were stalled on memory) at or above which a run at its limit is held there.