morca

morca · API

orca.pressure

Memory pressure (#202): Linux's pressure stall information (PSI) for memory, read each tick, and the admission rule built on it.

some avg10=1.53 avg60=0.87 avg300=0.24 total=1234567 full avg10=0.00 avg60=0.12 avg300=0.03 total=34567

some is the share of the last 10, 60 and 300 seconds in which at least one task was stalled waiting for memory (reclaim, swap-in, refaults); full the share in which every runnable task was. Meta's TMO acts on the same signal before memory runs out (Weiner et al., ASPLOS 2022).

Which file (find_source, once per process):

  1. ORCA_MEMORY_PRESSURE_FILE, if set: that file, whatever it is (tests point it at a fixture);
  2. on Linux, the cgroup v2 memory.pressure of the nearest cgroup, from this process's own up, whose memory.max is a limit: in a Docker container run with --memory that's the container's own (/sys/fs/cgroup/memory.pressure), which stalls when the container reaches its limit even with the host at ease;
  3. the host's /proc/pressure/memory. Inside a container it is the host's (PSI isn't namespaced), which is what a container without a limit shares;
  4. the own cgroup's memory.pressure though it has no limit, for a kernel with cgroup PSI but no readable /proc/pressure;
  5. none: macOS, or a kernel built or booted without PSI (psi=0). Then nothing is held, and the scheduler says so once.

Admission (Admission): while some avg10, less what runs held at their own memory limits explain (the most of their cgroups' some avg10), is over the limit (ORCA_ADMIT_MEMORY_PRESSURE, percent, default 10; 0 turns it off), no new run starts on this host; runs already going go on. The hold ends once avg10 is under half the limit, or has stayed at most the limit for RESUME_SECONDS, so a reading that hovers at the limit doesn't flap. A hold that has lasted ORCA_ADMIT_PRESSURE_GRACE_SECONDS (600) with none of orca's runs going on this host lets one run start at a time (escape): the pressure isn't orca's, and orca shouldn't wait for it forever. The caller logs and alerts once when that happens (stuck).

This module reads files and decides; it keeps nothing in a database, so a step (memory_pressure, from ORCA_MEMORY_PRESSURE_FILE, which every run on the local runner, the control host and a worker is given) and the per-run cgroups of #201 can use it alone.

Structs

Stall

struct Stall

One line of a PSI file: the percent of time stalled, averaged over 10, 60 and 300 seconds, and the total stalled, in microseconds.

Implements Copyable, ImplicitlyCopyable, Movable, Writable

Fields
avg10: Float64
avg60: Float64
avg300: Float64
total: Int

zero

def zero() -> Self

Psi

struct Psi

A PSI file's some and full lines; has_full is False for a file without one (/proc/pressure/cpu before 5.13, or a fixture).

Implements Copyable, ImplicitlyCopyable, Movable, Writable

Fields
some: Stall
full: Stall
has_full: Bool

Source

struct Source

The PSI file a host's memory pressure is read from: kind is file (ORCA_MEMORY_PRESSURE_FILE), cgroup (a cgroup with a memory limit), host (/proc/pressure/memory), cgroup-unlimited, or empty with no path for none.

Implements Boolable, Copyable, Movable, Writable

Fields
path: String
kind: String
limit_bytes: Int: The cgroup's memory.max for cgroup; else 0.

__bool__

def __bool__(self) -> Bool

none

def none() -> Self

describe

def describe(self) -> String

In words, for a log line and Ops › Control.

Admission

struct Admission

Whether this host starts new runs, from each reading of its memory pressure (observe), with hysteresis. Times are seconds on any clock that only goes forward.

Implements Copyable, Movable

Fields
limit: Float64: some avg10 percent over which runs wait; 0 for off.
grace: Float64: How long a hold lasts, with none of orca's runs going, before one may start anyway.
holding: Bool
held_since: Float64
calm_since: Float64: While holding, when avg10 came to at most the limit; -1 if it isn't.
reading: Optional[Psi]: The newest reading; None without one.
hold: String: While holding: the reason a run waits (WAITING_PRESSURE...); else empty.
escape: Bool: Holding past grace with nothing of orca's going: one run may start while none is.
stuck: Bool: escape became true at this reading: log it and alert, once a hold.
released: Bool: A hold ended at this reading.

__init__

def __init__(out self, limit: Float64, grace: Float64 = 600)

from_env

def from_env() -> Self

observe

def observe(mut self, psi: Optional[Psi], now: Float64, idle: Bool, own: Float64 = 0)

Takes a reading (None: none could be read, which holds nothing), at now, with idle whether none of orca's runs is going on this host. own is the part of some avg10 that runs held at their own memory limits explain (orca.cgroup.own_limit_pressure), which holds nothing: the host may have memory to spare, and such a run is killed and retried bigger in its own time.

held

def held(self) -> Bool

allows

def allows(self, idle: Bool) -> Int

How many starts this host may make now: none while holding, but one while escape and idle; -1 for no limit from pressure.

Functions

parse_psi

def parse_psi(text: String) -> Psi

A PSI file's text; raises without a some line, or on a line that isn't some|full avg10=.. avg60=.. avg300=.. total=...

read_psi

def read_psi(path: String) -> Optional[Psi]

The PSI file at path, or None if it can't be read or parsed (no file, PSI off in the kernel, a fixture half written).

memory_pressure

def memory_pressure() -> Optional[Psi]

This run's host's memory pressure, from the file ORCA_MEMORY_PRESSURE_FILE names (which orca gives every run it starts where there is one), or None. A step can poll it and lower its own parallelism while some.avg10 is high.

own_cgroup

def own_cgroup(proc_self_cgroup: String) -> String

The cgroup v2 path in /proc/self/cgroup's text (0::/path), or empty for none (cgroup v1 only).

pick_source

def pick_source(override: String, cgroup_dirs: List[String], limits: List[Int], has_pressure: List[Bool], host_ok: Bool) -> Source

The choice find_source makes, from what it found: override (ORCA_MEMORY_PRESSURE_FILE), the cgroup directories from this process's own up to the root of the mount, each one's memory.max in bytes (0 for none) and whether it has a readable memory.pressure, and whether /proc/pressure/memory reads.

cgroup_dirs

def cgroup_dirs(rel: String, root: String = CGROUP_ROOT) -> List[String]

The directories under root of cgroup rel and its parents, own first, to root itself. With a private cgroup namespace (Docker's default on cgroup v2) rel is / and root is the container's cgroup.

find_source

def find_source() -> Source

Where this host's memory pressure is read from; see the module's docs. Reads files, so it's called once.

pct_text

def pct_text(x: Float64) -> String

A percent for a reason: whole from 10, else to a tenth without a trailing .0: 23, 7.5, 0.

admit_limit

def admit_limit() -> Float64

ORCA_ADMIT_MEMORY_PRESSURE, percent; 10 when unset, and 0 (off) for a value that isn't a number from 0.

grace_seconds

def grace_seconds() -> Float64

pressure_env

def pressure_env(s: Source) -> List[String]

ORCA_MEMORY_PRESSURE_FILE=<path> for a run's process, so a step can poll its host's pressure (memory_pressure); nothing without one.

Aliases

comptime PRESSURE_FILE = "ORCA_MEMORY_PRESSURE_FILE"

The file a run polls for its host's memory pressure, and the scheduler's override of which file it reads.

comptime ADMIT_LIMIT = "ORCA_ADMIT_MEMORY_PRESSURE"

The some avg10 percent over which no new run starts; 0 turns it off.

comptime DEFAULT_LIMIT = SIMD(10)
comptime GRACE = "ORCA_ADMIT_PRESSURE_GRACE_SECONDS"
comptime DEFAULT_GRACE = SIMD(600)
comptime RESUME_SECONDS = SIMD(30)

How long avg10 stays at most the limit before a hold ends anyway.

comptime WAITING_PRESSURE = "waiting: memory pressure "

How a run held by memory pressure's reason starts.

comptime HOST_PSI = "/proc/pressure/memory"
comptime CGROUP_ROOT = "/sys/fs/cgroup"