morca

morca · API

orca.metrics

A run's machine metrics on its page: the samplers' files (orca.sample) read back, their peaks and warnings, and the charts, drawn as SVG on the server like the DAG.

read_metrics reads every <ORCA_LOGS>/<pipeline>/<run>/metrics/*.jsonl, local or in object storage, into one InstanceMetrics per file; a line it can't read is skipped. metrics_svg draws them under a lane of the run's steps, every row on the same time axis (wall-clock time, from the first attempt or sample to the last), so a step that pegged a CPU or filled a disk lines up with its bar:

  • cpu: user and system time as an area, iowait above it, and steal as a line, in percent of all cores;
  • mem · disk: memory used and the fullest disk's used space, in percent of the instance's;
  • io · net: disk reads and writes and network traffic, in bytes per second, each scaled to the instance's highest rate, which the label says.

A warning is a stretch of samples where memory was over 90% of the instance's, a disk was over 90% full or CPU steal was over 20%, since those explain most slow or failed runs. Each is drawn as a band over the steps lane and the instance's rows, and listed under the peaks.

Structs

Sample

struct Sample

Implements Copyable, Movable

Fields
t: Float64: Unix time.
dt: Float64
user: Float64
system: Float64
iowait: Float64
steal: Float64
busiest: Float64
mem_total: Int
mem_used: Int
disk_total: Int: Of the fullest disk.
disk_used: Int
io: Int: Disk bytes read and written per second; -1 where not measured.
net: Int

cpu

def cpu(self) -> Float64

mem_pct

def mem_pct(self) -> Float64

disk_pct

def disk_pct(self) -> Float64

Warning

struct Warning

Implements Copyable, Movable

Fields
kind: String: memory, disk or steal.
start: Float64
end: Float64
peak: Float64: The highest percentage in it.

what

def what(self) -> String

Peaks

struct Peaks

Implements Copyable, Movable

Fields
cpu: Float64: User and system, percent of all cores.
busiest: Float64
steal: Float64
mem_used: Int
mem_total: Int
mem_pct: Float64
disk_used: Int
disk_total: Int
disk_pct: Float64
io: Int
net: Int

InstanceMetrics

struct InstanceMetrics

Implements Copyable, Movable

Fields
name: String: The file's name without .jsonl: local, <role>-<instance id> or a pod's name.
samples: List[Sample]
skipped: Int: Lines that weren't samples.

peaks

def peaks(self) -> Peaks

warnings

def warnings(self) -> List[Warning]

Stretches of samples over a threshold, each from the start of its first sample's interval to its last sample.

StepSpan

struct StepSpan

An attempt, as the steps lane draws it.

Implements Copyable, Movable

Fields
step: String
partition: String
status: String
start: Float64
end: Float64
attempt: Int: Its ID, which shows its card (orca.ui.step_cards), or 0.

Functions

parse_sample

def parse_sample(line: String) -> Sample

A sampler's line; raises if it isn't one.

parse_samples

def parse_samples(name: String, text: String) -> InstanceMetrics

A sampler's file, in time order; lines that aren't samples are counted and skipped.

read_metrics

def read_metrics(pipeline: String, run: Int) -> List[InstanceMetrics]

Every instance's samples for the run under ORCA_LOGS; none without it, or if the directory can't be listed.

bytes_text

def bytes_text(n: Int) -> String

n bytes as 512 B, 3.2 MB, 12 GB.

pct_text

def pct_text(p: Float64) -> String

peaks_json

def peaks_json(m: InstanceMetrics) -> String

An instance's peaks and warnings, for the run's JSON.

axis_of

def axis_of(instances: List[InstanceMetrics], steps: List[StepSpan]) -> Tuple[Float64, Float64]

The axis's start and end: the first attempt or sample to the last.

metrics_svg

def metrics_svg(instances: List[InstanceMetrics], steps: List[StepSpan], live: Bool, label: String = "The run's machines over time") -> String

The steps lane, then three strips per instance, on one time axis, with the warnings as bands; without instances, the steps lane alone. label names it for screen readers.

metrics_html

def metrics_html(mut h: Html, instances: List[InstanceMetrics], steps: List[StepSpan], live: Bool)

The charts, a legend, then each instance's peaks and warnings.

Aliases

comptime MAX_SAMPLES = 50000

Samples read per instance at most: 69 hours at 5 s.

comptime MEMORY_WARN = SIMD(90)
comptime DISK_WARN = SIMD(90)
comptime STEAL_WARN = SIMD(20)