orca.usageWhat a run's processes use: each attempt's peak memory and CPU.
Whoever starts a run's process measures it: the scheduler for a local or
control-placed run (orca.scheduler), orca worker for a remote worker's
(orca.worker). Each tick, at most every quarter of a second, it reads
the process's tree in /proc (orca.sample.tree_usage: the driver and what
it started, dista's local workers among them) and keeps a Meter of it:
the resident bytes, the CPU seconds so far (utime + stime, with what the
tree's processes waited for), and the cores that kept busy since the
sample before. When the process is reaped, wait4 adds its own word:
ru_maxrss, the largest single process (a lower bound on the tree's), and
its CPU seconds, which count descendants that ended after the last sample.
A process runs every attempt of a run's start, so its samples are shared
out among those attempts by time (attribute): an attempt's peak is the
largest sample taken while it ran (or, for one shorter than a tick, the
sample just after it), its CPU the meter's CPU at its end less at its
start, its peak cores the busiest interval it overlapped. ru_maxrss
raises the peak of a process's only attempt. A meter keeps at most
MAX_MARKS samples, merging neighbours as it fills (keeping their maximum
and the later total), so a long run's samples are coarser, not dropped.
On macOS, which has no /proc, there are no samples: a process's only
attempt gets wait4's numbers, and none gets its peak cores; a process
of several attempts records none. Runs on EC2 or Kubernetes, and runs no
scheduler or worker started (<binary> run by hand), record nothing:
their columns stay NULL.
The numbers are orca.attempts' peak_rss_bytes, cpu_seconds and
peak_cpus (migration 35), written once, when the process ends: by the
scheduler over Postgres, and for a worker's run by orca ui, which the
worker sends its meter with /api/worker/exited (marks_json, ages in
seconds before the request, so neither clock needs to agree with the
database's).
Markstruct MarkA sample of a process tree. at is seconds on the monotonic clock in a Meter, and seconds before now once aged.
Implements Copyable, ImplicitlyCopyable, Movable
at: Float64rss: Int: The most resident bytes sampled up to here since the mark before; -1 for none (the start, and the end).cpu: Float64: The CPU seconds so far.cpus: Float64: The most cores busy over a sample's interval since the mark before; -1 for none.Meterstruct MeterOne run process's samples, from its start to its end.
Implements Copyable, Movable
marks: List[Mark]max_rss: Int: wait4's ru_maxrss in bytes once it has ended, else -1.__init__def __init__(out self, now: Float64 = Float64("-1"))sampledef sample(mut self, rss: Int, cpu: Float64, now_in: Float64 = Float64("-1"))Adds a reading of the tree: its resident bytes and CPU seconds.
enddef end(mut self, max_rss: Int, cpu: Float64, now_in: Float64 = Float64("-1"))The process ended, wait4 says having used cpu seconds with max_rss bytes its largest process.
ageddef aged(self, now_in: Float64 = Float64("-1")) -> List[Mark]The marks with at seconds before now, oldest first.
AttemptSpanstruct AttemptSpanAn attempt the process made, in seconds before now.
Implements Copyable, ImplicitlyCopyable, Movable
id: Intstarted: Float64ended: Float64: -1 if it hadn't ended when the process did.attributedef attribute(marks: List[Mark], max_rss: Int, attempts: List[AttemptSpan]) -> List[Usage]Each attempt's share of a process's marks (aged, oldest first, the start first and the end last), as described above.
recorddef record(mut db: State, run: Int, pid: Int, marks: List[Mark], max_rss: Int, worker: Int = Int(0), start: Int = Int(0)) -> IntWrites each attempt's share of the process pid's marks (aged at about the database's now) to the attempts of run that it made, and returns how many. With worker, only while the run's start start is that worker's.
marks_jsondef marks_json(marks: List[Mark]) -> StringAged marks as [[age, rss, cpu, cpus], ...].
marks_ofdef marks_of(v: Value) -> List[Mark]marks_json's marks back.
bytes_textdef bytes_text(n: Int) -> StringResident bytes as MB (MiB, as ORCA_CONTROL_MEMORY_MB counts them), or GB from 10 GB.
cpu_textdef cpu_text(s: Float64) -> StringCPU seconds, as seconds, minutes or hours.
usage_textdef usage_text(u: Usage) -> Stringpeak 812 MB, CPU 41.2 s on up to 3.9 cores, as much as is known; empty for nothing.
usage_ofdef usage_of(v: Value) -> Usageusage_json's usage back; -1 for what's null or missing.
usage_jsondef usage_json(u: Usage) -> String{"peak_rss_bytes": ..., "cpu_seconds": ..., "peak_cpus": ...}, null for what isn't known.
comptime MAX_MARKS = 1024The most samples a meter keeps; past it, neighbours are merged.
comptime MIN_SAMPLE_SECONDS = 0.25The least time between two samples of one process.