# orca.sample > A module of morca's API. The sampler: a run's machine metrics, one JSON line every few seconds. ``` orca sample --out [--pid PID] [--until FILE] [--match PATH] [--dirs D,...] [--seconds S] [--count N] [--keep N] [--cgroup DIR] [--names DIR] sample ... the same, from a pipeline binary ``` It runs beside every process that does a run's work, started by the runner: on the ec2 runner by each instance's user data, next to dista (`orca.ec2`); on the local runner beside each run's process (`orca.scheduler`); on the kube runner in each pod (`orca.kube`). The runners start it as ` sample`, since the pipeline's binary is what every instance and image has; `orca sample` is the same command. Every `--seconds` (`ORCA_SAMPLE_SECONDS`, default 5; the first line a fifth of a second after it starts, so any instance it ran on leaves one) it reads `/proc` on Linux, or sysctls and the Mach host calls on macOS, and appends a line: ``` {"v": 1, "t": 1762250000.123, "dt": 5.0, "host": "ip-10-0-1-5", "cores": 4, "cpu": {"user": 61.2, "system": 4.1, "iowait": 0.3, "steal": 0.0, "busiest": 99.0}, "mem": {"total": 8053063680, "used": 3221225472, "available": 4831838208, "cache": 2147483648, "swap_total": 0, "swap_used": 0, "rss": 2952790016}, "disk": [{"path": "/tmp", "total": 53687091200, "used": 9663676416, "free": 44023414784}], "io": {"read": 1048576, "write": 52428800}, "net": {"in": 1310720, "out": 655360}, "containers": [{"name": "postgres", "mem": 412000000}], "self_cpu": 0.42} ``` - `t` is the Unix time, `dt` the seconds since the previous line. - `cpu`: percentages of all cores' time over `dt`; `busiest` is the busiest core's. `iowait` and `steal` are always 0 on macOS. - `mem`: bytes. `used` is `total - available` on Linux (`MemAvailable`), and app, wired and compressed memory on macOS; `cache` is the page cache (file-backed pages). `rss` is the resident set of the run's own processes: `--pid` and its descendants, and on Linux every process whose command is `--match` (on the ec2 runner, the binary in its containers, which aren't the script's children). - `disk`: per directory the run writes to (`--dirs`, default `TMPDIR` or `/tmp`, and dista's `DISTA_SPILL_DIR` and `DISTA_STORE_DIR` when set), one per file system, in bytes; `free` is what an unprivileged process can use. - `io` and `net`: bytes per second over `dt`, summed over whole disks and over interfaces other than loopback. `io` is `null` on macOS. - `containers`: only with `--cgroup`; see below. - `self_cpu`: the sampler's own CPU seconds so far, for its overhead. Later fields may be added (CPU counters), so readers ignore keys they don't know; `v` changes only if one that's here changes meaning. `--out` is where the lines go: a file, a location in object storage (lines are appended to its `orca.logs.stage_of` staging file and the whole of it uploaded every `ORCA_SAMPLE_UPLOAD_SECONDS`, 30 by default, and when the sampler ends), or `-`, standard output with each line prefixed `orca-sample `, for a pod whose log the scheduler keeps. The runners put it at `///metrics/.jsonl` (`metrics_location`). `--keep N` keeps a file `--out` names to about its last N lines, for a sampler that never ends (the control instance's, which the scheduler reads back into the fleet page, `orca.fleet`): once the file has 2 N it's rewritten with its last N, atomically, so a reader sees either. `--cgroup DIR` (the control instance's sampler) adds `containers`: the memory of each Docker container, in bytes, as `docker stats` counts it (`memory.current` less the page cache's inactive files), read from the cgroup files under `DIR`, the host's `/sys/fs/cgroup` mounted read-only (`system.slice/docker-.scope` with the systemd driver, `docker/` with cgroupfs; cgroup v2 only). `--names DIR` is the host's `/var/lib/docker/containers`, also read-only, whose `/config.v2.json` gives a container its name; without it a container is its first 12 characters of id. Reading files, not the Docker socket, keeps the sampler unprivileged: the socket is root on the host, these mounts are not. It ends after a last line once the `--pid` process has, or once the file `--until` names exists, or after `--count` lines. SIGTERM ends it at once, keeping the lines it wrote (Mojo's runtime threads take signals a blocked main thread wouldn't, so it can't wait for one), which is why the runners stop it with `--pid` or `--until`. A line it can't write is dropped with one message, and it carries on: a sampler never fails a run. ## Structs ### `PidUsage` ```mojo struct PidUsage ``` What /proc//stat says one process uses. Implements: `Copyable`, `ImplicitlyCopyable`, `Movable` Fields: - `parent: Int`: -1 if the line was malformed. - `rss_pages: Int` - `cpu_ticks: Int`: utime + stime + cutime + cstime: its own CPU and that of the children it waited for, in clock ticks. ### `CpuShare` ```mojo struct CpuShare ``` Percentages of all cores' time between two readings. Implements: `Copyable`, `Movable` Fields: - `user: Float64` - `system: Float64` - `iowait: Float64` - `steal: Float64` - `busiest: Float64` ### `Disk` ```mojo struct Disk ``` Implements: `Copyable`, `Movable` Fields: - `path: String` - `total: Int` - `used: Int` - `free: Int` - `fsid: Int` ### `Mem` ```mojo struct Mem ``` Implements: `Copyable`, `Movable` Fields: - `total: Int` - `used: Int` - `available: Int` - `cache: Int` - `swap_total: Int` - `swap_used: Int` ### `Sampler` ```mojo struct Sampler ``` The readings a line is the difference of, and what to read. Implements: `Movable` Fields: - `root: Int` - `argv0: String` - `dirs: List[String]` - `host: String` - `cores: Int` - `disks: List[String]`: Linux's whole disks, from /sys/block. - `task: UInt32`: macOS: this task's port, to free what host_processor_info returns. - `cpu: List[List[Int]]` - `io: Tuple[Int, Int]` - `net: Dict[String, Tuple[Int, Int]]` - `net_total: Tuple[Int, Int]` - `at: Int`: perf_counter_ns of the readings. - `cgroup: String`: `--cgroup`: where the containers' cgroups are; empty for none. - `names: String`: `--names`: where the containers' config files are. - `known: Dict[String, String]`: Container ids and the names read for them. #### `__init__` ```mojo def __init__(out self, root: Int, argv0: String, var dirs: List[String], cgroup: String = "", names: String = "") ``` #### `line` ```mojo def line(mut self) -> String ``` A JSON line covering the time since the last one. ## Functions ### `sample_seconds` ```mojo def sample_seconds() -> Float64 ``` `ORCA_SAMPLE_SECONDS`: the sampler's interval; 0 turns it off. A value that isn't a number is the default, 5. ### `metrics_location` ```mojo def metrics_location(pipeline: String, run: Int, instance: String) -> String ``` Where an instance's samples go under `ORCA_LOGS`; "" without it. ### `machine_name` ```mojo def machine_name(name: String) -> String ``` The worker's name as its runs' metrics file names it (`metrics/worker-.jsonl`): `worker-` and the name, with anything but letters, digits, `.`, `-` and `_` made `_`. ### `parse_proc_stat` ```mojo def parse_proc_stat(text: String) -> List[List[Int]] ``` /proc/stat's `cpu` lines: all cores first, then each core, as user, nice, system, idle, iowait, irq, softirq and steal ticks. ### `parse_meminfo` ```mojo def parse_meminfo(text: String) -> Dict[String, Int] ``` /proc/meminfo, in bytes. ### `parse_diskstats` ```mojo def parse_diskstats(text: String, disks: List[String]) -> Tuple[Int, Int] ``` Bytes read and written, summed over the whole disks named in `disks` (all but `_skipped_disk` ones if it's empty, which counts partitions twice). ### `parse_net_dev` ```mojo def parse_net_dev(text: String) -> Dict[String, Tuple[Int, Int]] ``` /proc/net/dev: bytes received and sent per interface, but loopback. ### `parse_pid_stat` ```mojo def parse_pid_stat(text: String) -> Tuple[Int, Int] ``` /proc//stat's parent and resident pages; (-1, 0) if malformed. ### `parse_pid_usage` ```mojo def parse_pid_usage(text: String) -> PidUsage ``` /proc//stat's parent, resident pages and CPU ticks. ### `cpu_share` ```mojo def cpu_share(prev: List[List[Int]], cur: List[List[Int]]) -> CpuShare ``` The share of each kind of CPU time between `prev` and `cur`, readings as `parse_proc_stat` gives them. ### `fixed1` ```mojo def fixed1(x: Float64) -> String ``` `x` to one decimal, as JSON. ### `alive` ```mojo def alive(pid: Int) -> Bool ``` Whether a process `pid` is running: one that has exited is dead, though its parent hasn't reaped it yet (a shell running the sampler in front doesn't reap the job it watches until the sampler ends). ### `container_mem` ```mojo def container_mem(current: String, stat: String) -> Int ``` A container's memory as `docker stats` counts it: a cgroup v2 `memory.current` less `inactive_file` from `memory.stat`; -1 if `current` isn't a number. ### `container_name` ```mojo def container_name(config: String) -> String ``` The name in a container's `config.v2.json` (`/postgres`), without the slash; empty if there is none. ### `container_ids` ```mojo def container_ids(root: String) -> List[Tuple[String, String]] ``` The Docker containers' cgroups under `root`, as (id, directory): the systemd driver's `system.slice/docker-.scope` and cgroupfs's `docker/`. ### `tree_rss` ```mojo def tree_rss(root: Int, argv0: String = "") -> Int ``` The resident bytes of `root`'s process tree, and on Linux of the processes whose command is `argv0`; the scheduler checks a control-placed run's against its cap with it (orca.control). ### `tree_usage` ```mojo def tree_usage(roots: List[Int]) -> Dict[Int, Tuple[Int, Float64]] ``` Each root's process tree's resident bytes and CPU seconds (its own and its descendants', with what they waited for: so a child's time stays counted once it ends), from one pass over /proc; the scheduler and `orca worker` sample their runs with it each tick (orca.usage). Empty on macOS, which has no /proc: there the runs' usage is what `wait4` says at their end. ### `default_dirs` ```mojo def default_dirs() -> List[String] ``` The directories a run writes to: the temporary one and dista's. ### `append_line` ```mojo def append_line(path: String, line: String) ``` Appends `line` and a newline to the file at `path`. ### `sample_main` ```mojo def sample_main(args: List[String]) -> Int ``` `sample`, with `args[2:]` its arguments. Never raises: whatever goes wrong, the run it watches goes on. ## Aliases ### `PREFIX` ```mojo comptime PREFIX = "orca-sample " ``` What starts a sample line written to standard output (`--out -`).