morca

morca · API

orca.system

The control instance's own health: /ops/control, the gauges on /metrics, and the warning on the home page.

The control instance (Postgres, the scheduler, the UI, Caddy, the apps) runs a sampler of its own: a container the stack's user data starts beside the scheduler, with the host's process and network namespaces and the host's cgroup files and Docker's config files mounted read-only (orca.sample --cgroup and --names). It keeps its last hour of lines in the file ORCA_CONTROL_SAMPLES names. Every ORCA_FLEET_SECONDS the scheduler reads the file (orca.fleet.ingest) and, here:

  • rolls the lines it hasn't seen into one row a minute in orca.system_minutes (CPU, memory, the fullest disk, network: sums to average over, and peaks), and one row per container and minute in orca.system_containers (its peak memory), kept KEEP_DAYS (14) days. "Hasn't seen" is the newest sample already in a row (last_t), so a pass again adds nothing;
  • appends those lines, as they are, to a file per UTC day, <ORCA_LOGS>/control/<date>.jsonl, so the history outlives the database and can be read back whole. In object storage a file is uploaded whole, every ORCA_CONTROL_UPLOAD_SECONDS (300) and once its day is over; the stack's bucket expires logs/control/ after 14 days, and a local ORCA_LOGS has files older than that removed.

A page view reads only those two tables: no file, no bucket. The gauges on /metrics are the newest minute's, left out once it is older than ten minutes, since a gauge that stays at its last value is worse than none.

Structs

Container

struct Container

Implements Copyable, Movable

Fields
name: String
mem: Int: Bytes, as docker stats counts them.

ControlSample

struct ControlSample

Implements Copyable, Movable

Fields
line: String
t: Float64
cpu: Float64: User and system, percent of all cores.
mem_pct: Float64
mem_used: Int
mem_total: Int
disk_pct: Float64: Of the fullest disk.
disk_used: Int
disk_total: Int
disk_path: String
net_in: Float64: Bytes per second.
net_out: Float64
containers: List[Container]

Minute

struct Minute

What the samples of one minute add up to.

Implements Copyable, Movable

Fields
minute: Int: Unix time of its start.
n: Int
last_t: Float64
cpu_sum: Float64
cpu_max: Float64
mem_sum: Float64
mem_max: Float64
mem_used: Int
mem_total: Int
disk_max: Float64
disk_used: Int
disk_total: Int
disk_path: String
net_in_sum: Float64
net_out_sum: Float64
names: List[String]
peaks: List[Int]: Each container's highest memory in the minute.

Archive

struct Archive

The day files, between passes: which ones have lines not yet uploaded, and when the last upload was.

Implements Movable

Fields
dirty: List[String]
next: Int: When the next upload may start, on the monotonic clock.
seconds: Float64

__init__

def __init__(out self)

Now

struct Now

The newest minute.

Implements Copyable, Movable

Fields
age: Float64: Seconds since its last sample.
cpu: Float64
cpu_peak: Float64
mem_pct: Float64
mem_used: Int
mem_total: Int
disk_pct: Float64
disk_used: Int
disk_total: Int
disk_path: String
net_in: Float64
net_out: Float64
containers: List[Container]

stale

def stale(self) -> Bool

Charts

struct Charts

Implements Copyable, Movable

Fields
hours: Int
bucket: Int: Seconds in a point.
labels: List[String]: MM-DD HH:MM (UTC) of each point.
cpu: List[Series]
mem: List[Series]
disk: List[Series]
net: List[Series]
containers: List[Series]: Each container's highest memory in MB.

__init__

def __init__(out self, hours: Int, bucket: Int, var labels: List[String], var cpu: List[Series], var mem: List[Series], var disk: List[Series], var net: List[Series], var containers: List[Series])

Functions

parse_control_sample

def parse_control_sample(line: String) -> ControlSample

A sampler's line; raises if it isn't one.

samples_after

def samples_after(text: String, after: Float64, n: Int = Int(800)) -> List[ControlSample]

The samples among the last n lines of the control instance's file that are newer than after (Unix time), oldest first; lines that aren't samples are skipped.

roll

def roll(samples: List[ControlSample]) -> List[Minute]

The samples, in time order, as one Minute each.

day_of

def day_of(t: Float64) -> String

The UTC date of Unix time t, YYYY-MM-DD.

day_file

def day_file(date: String) -> String

Where the day's lines go, or "" without ORCA_LOGS.

external_time

def external_time() -> Int

Unix time now.

keep

def keep(mut db: State, text: String, mut archive: Archive) -> Int

Rolls the new samples in text (the control instance's file) into the database and the day files; returns how many were new.

trim

def trim(mut db: State)

Drops what is older than KEEP_DAYS, in the database and in a local ORCA_LOGS.

system_now

def system_now(mut db: State) -> Optional[Now]

The newest minute the control instance sampled, or None if it never has. Check stale(): it may be long ago.

warnings

def warnings(now: Optional[Now]) -> List[String]

What is wrong with the control instance now, in words: a disk over DISK_WARN percent or memory over MEMORY_WARN, since a full disk stops every run. Nothing for samples that are old or missing.

alert_html

def alert_html(mut h: Html, w: List[String])

The warnings as a box linking to /ops/control, or nothing.

warnings_json

def warnings_json(w: List[String]) -> String

range_hours

def range_hours(range: String) -> Int

The hours ?range= asks for: 24, 168 (7d) or 336 (14d).

system_charts

def system_charts(mut db: State, hours: Int) -> Charts

The minutes in the last hours, in at most 336 points: every 5 minutes for a day, 30 for a week, an hour for two weeks. A point with no samples is 0.

system_html

def system_html(mut h: Html, now: Optional[Now], c: Charts, range: String)

The cards, the containers' table and the charts.

now_json

def now_json(now: Optional[Now]) -> String

system_json

def system_json(now: Optional[Now], c: Charts) -> String

prometheus

def prometheus(now: Optional[Now], f: Summary) -> String

The gauges, in Prometheus' text format: the fleet's (f, from orca.fleet) and the control instance's, which are left out while its samples are stale or missing.

Aliases

comptime KEEP_DAYS = 14

How long the minutes, the containers' and the day files are kept.

comptime DISK_WARN = SIMD(85)

The fullest disk's percent over which the home page warns.

comptime MEMORY_WARN = SIMD(90)

Memory's percent over which the home page warns.

comptime FRESH_SECONDS = 600

How old the newest sample may be and still be what's happening now.