orca.sample

orca sample --out <location> [--pid PID] [--until FILE] [--match PATH]
            [--dirs D,...] [--seconds S] [--count N] [--keep N]
            [--cgroup DIR] [--names DIR]
<pipeline> sample ...                       the same, from a pipeline binary

It runs beside every process that does a run's work, started by the runner: on the ec2 runner by each instance's user data, next to dista (orca.ec2); on the local runner beside each run's process (orca.scheduler); on the kube runner in each pod (orca.kube). The runners start it as <binary> sample, since the pipeline's binary is what every instance and image has; orca sample is the same command.

Every --seconds (ORCA_SAMPLE_SECONDS, default 5; the first line a fifth of a second after it starts, so any instance it ran on leaves one) it reads /proc on Linux, or sysctls and the Mach host calls on macOS, and appends a line:

{"v": 1, "t": 1762250000.123, "dt": 5.0, "host": "ip-10-0-1-5", "cores": 4,
 "cpu": {"user": 61.2, "system": 4.1, "iowait": 0.3, "steal": 0.0, "busiest": 99.0},
 "mem": {"total": 8053063680, "used": 3221225472, "available": 4831838208,
         "cache": 2147483648, "swap_total": 0, "swap_used": 0, "rss": 2952790016},
 "disk": [{"path": "/tmp", "total": 53687091200, "used": 9663676416, "free": 44023414784}],
 "io": {"read": 1048576, "write": 52428800}, "net": {"in": 1310720, "out": 655360},
 "containers": [{"name": "postgres", "mem": 412000000}], "self_cpu": 0.42}

• t is the Unix time, dt the seconds since the previous line.
• cpu: percentages of all cores' time over dt; busiest is the busiest core's. iowait and steal are always 0 on macOS.
• mem: bytes. used is total - available on Linux (MemAvailable), and app, wired and compressed memory on macOS; cache is the page cache (file-backed pages). rss is the resident set of the run's own processes: --pid and its descendants, and on Linux every process whose command is --match (on the ec2 runner, the binary in its containers, which aren't the script's children).
• disk: per directory the run writes to (--dirs, default TMPDIR or /tmp, and dista's DISTA_SPILL_DIR and DISTA_STORE_DIR when set), one per file system, in bytes; free is what an unprivileged process can use.
• io and net: bytes per second over dt, summed over whole disks and over interfaces other than loopback. io is null on macOS.
• containers: only with --cgroup; see below.
• self_cpu: the sampler's own CPU seconds so far, for its overhead.

Later fields may be added (CPU counters), so readers ignore keys they don't know; v changes only if one that's here changes meaning.

--out is where the lines go: a file, a location in object storage (lines are appended to its orca.logs.stage_of staging file and the whole of it uploaded every ORCA_SAMPLE_UPLOAD_SECONDS, 30 by default, and when the sampler ends), or -, standard output with each line prefixed orca-sample , for a pod whose log the scheduler keeps. The runners put it at <ORCA_LOGS>/<pipeline>/<run>/metrics/<instance>.jsonl (metrics_location).

--keep N keeps a file --out names to about its last N lines, for a sampler that never ends (the control instance's, which the scheduler reads back into the fleet page, orca.fleet): once the file has 2 N it's rewritten with its last N, atomically, so a reader sees either.

--cgroup DIR (the control instance's sampler) adds containers: the memory of each Docker container, in bytes, as docker stats counts it (memory.current less the page cache's inactive files), read from the cgroup files under DIR, the host's /sys/fs/cgroup mounted read-only (system.slice/docker-<id>.scope with the systemd driver, docker/<id> with cgroupfs; cgroup v2 only). --names DIR is the host's /var/lib/docker/containers, also read-only, whose <id>/config.v2.json gives a container its name; without it a container is its first 12 characters of id. Reading files, not the Docker socket, keeps the sampler unprivileged: the socket is root on the host, these mounts are not.

It ends after a last line once the --pid process has, or once the file --until names exists, or after --count lines. SIGTERM ends it at once, keeping the lines it wrote (Mojo's runtime threads take signals a blocked main thread wouldn't, so it can't wait for one), which is why the runners stop it with --pid or --until. A line it can't write is dropped with one message, and it carries on: a sampler never fails a run.

struct PidUsage

struct CpuShare

struct Disk

struct Mem

struct Sampler

def __init__(out self, root: Int, argv0: String, var dirs: List[String], cgroup: String = "", names: String = "")

def line(mut self) -> String

def sample_seconds() -> Float64

def metrics_location(pipeline: String, run: Int, instance: String) -> String

def machine_name(name: String) -> String

def parse_proc_stat(text: String) -> List[List[Int]]

def parse_meminfo(text: String) -> Dict[String, Int]

def parse_diskstats(text: String, disks: List[String]) -> Tuple[Int, Int]

def parse_net_dev(text: String) -> Dict[String, Tuple[Int, Int]]

def parse_pid_stat(text: String) -> Tuple[Int, Int]

def parse_pid_usage(text: String) -> PidUsage

def cpu_share(prev: List[List[Int]], cur: List[List[Int]]) -> CpuShare

def fixed1(x: Float64) -> String

def alive(pid: Int) -> Bool

def container_mem(current: String, stat: String) -> Int

def container_name(config: String) -> String

def container_ids(root: String) -> List[Tuple[String, String]]

def tree_rss(root: Int, argv0: String = "") -> Int

def tree_usage(roots: List[Int]) -> Dict[Int, Tuple[Int, Float64]]

def default_dirs() -> List[String]

def append_line(path: String, line: String)

def sample_main(args: List[String]) -> Int

comptime PREFIX = "orca-sample "