# orca.system > A module of morca's API. The control instance's own health: `/ops/control`, the gauges on `/metrics`, and the warning on the home page. The control instance (Postgres, the scheduler, the UI, Caddy, the apps) runs a sampler of its own: a container the stack's user data starts beside the scheduler, with the host's process and network namespaces and the host's cgroup files and Docker's config files mounted read-only (`orca.sample` `--cgroup` and `--names`). It keeps its last hour of lines in the file `ORCA_CONTROL_SAMPLES` names. Every `ORCA_FLEET_SECONDS` the scheduler reads the file (`orca.fleet.ingest`) and, here: - rolls the lines it hasn't seen into one row a minute in `orca.system_minutes` (CPU, memory, the fullest disk, network: sums to average over, and peaks), and one row per container and minute in `orca.system_containers` (its peak memory), kept `KEEP_DAYS` (14) days. "Hasn't seen" is the newest sample already in a row (`last_t`), so a pass again adds nothing; - appends those lines, as they are, to a file per UTC day, `/control/.jsonl`, so the history outlives the database and can be read back whole. In object storage a file is uploaded whole, every `ORCA_CONTROL_UPLOAD_SECONDS` (300) and once its day is over; the stack's bucket expires `logs/control/` after 14 days, and a local `ORCA_LOGS` has files older than that removed. A page view reads only those two tables: no file, no bucket. The gauges on `/metrics` are the newest minute's, left out once it is older than ten minutes, since a gauge that stays at its last value is worse than none. ## Structs ### `Container` ```mojo struct Container ``` Implements: `Copyable`, `Movable` Fields: - `name: String` - `mem: Int`: Bytes, as `docker stats` counts them. ### `ControlSample` ```mojo struct ControlSample ``` Implements: `Copyable`, `Movable` Fields: - `line: String` - `t: Float64` - `cpu: Float64`: User and system, percent of all cores. - `mem_pct: Float64` - `mem_used: Int` - `mem_total: Int` - `disk_pct: Float64`: Of the fullest disk. - `disk_used: Int` - `disk_total: Int` - `disk_path: String` - `net_in: Float64`: Bytes per second. - `net_out: Float64` - `containers: List[Container]` ### `Minute` ```mojo struct Minute ``` What the samples of one minute add up to. Implements: `Copyable`, `Movable` Fields: - `minute: Int`: Unix time of its start. - `n: Int` - `last_t: Float64` - `cpu_sum: Float64` - `cpu_max: Float64` - `mem_sum: Float64` - `mem_max: Float64` - `mem_used: Int` - `mem_total: Int` - `disk_max: Float64` - `disk_used: Int` - `disk_total: Int` - `disk_path: String` - `net_in_sum: Float64` - `net_out_sum: Float64` - `names: List[String]` - `peaks: List[Int]`: Each container's highest memory in the minute. ### `Archive` ```mojo struct Archive ``` The day files, between passes: which ones have lines not yet uploaded, and when the last upload was. Implements: `Movable` Fields: - `dirty: List[String]` - `next: Int`: When the next upload may start, on the monotonic clock. - `seconds: Float64` #### `__init__` ```mojo def __init__(out self) ``` ### `Now` ```mojo struct Now ``` The newest minute. Implements: `Copyable`, `Movable` Fields: - `age: Float64`: Seconds since its last sample. - `cpu: Float64` - `cpu_peak: Float64` - `mem_pct: Float64` - `mem_used: Int` - `mem_total: Int` - `disk_pct: Float64` - `disk_used: Int` - `disk_total: Int` - `disk_path: String` - `net_in: Float64` - `net_out: Float64` - `containers: List[Container]` #### `stale` ```mojo def stale(self) -> Bool ``` ### `Charts` ```mojo struct Charts ``` Implements: `Copyable`, `Movable` Fields: - `hours: Int` - `bucket: Int`: Seconds in a point. - `labels: List[String]`: `MM-DD HH:MM` (UTC) of each point. - `cpu: List[Series]` - `mem: List[Series]` - `disk: List[Series]` - `net: List[Series]` - `containers: List[Series]`: Each container's highest memory in MB. #### `__init__` ```mojo def __init__(out self, hours: Int, bucket: Int, var labels: List[String], var cpu: List[Series], var mem: List[Series], var disk: List[Series], var net: List[Series], var containers: List[Series]) ``` ## Functions ### `parse_control_sample` ```mojo def parse_control_sample(line: String) -> ControlSample ``` A sampler's line; raises if it isn't one. ### `samples_after` ```mojo def samples_after(text: String, after: Float64, n: Int = Int(800)) -> List[ControlSample] ``` The samples among the last `n` lines of the control instance's file that are newer than `after` (Unix time), oldest first; lines that aren't samples are skipped. ### `roll` ```mojo def roll(samples: List[ControlSample]) -> List[Minute] ``` The samples, in time order, as one `Minute` each. ### `day_of` ```mojo def day_of(t: Float64) -> String ``` The UTC date of Unix time `t`, `YYYY-MM-DD`. ### `day_file` ```mojo def day_file(date: String) -> String ``` Where the day's lines go, or "" without `ORCA_LOGS`. ### `external_time` ```mojo def external_time() -> Int ``` Unix time now. ### `keep` ```mojo def keep(mut db: State, text: String, mut archive: Archive) -> Int ``` Rolls the new samples in `text` (the control instance's file) into the database and the day files; returns how many were new. ### `trim` ```mojo def trim(mut db: State) ``` Drops what is older than `KEEP_DAYS`, in the database and in a local `ORCA_LOGS`. ### `system_now` ```mojo def system_now(mut db: State) -> Optional[Now] ``` The newest minute the control instance sampled, or None if it never has. Check `stale()`: it may be long ago. ### `warnings` ```mojo def warnings(now: Optional[Now]) -> List[String] ``` What is wrong with the control instance now, in words: a disk over `DISK_WARN` percent or memory over `MEMORY_WARN`, since a full disk stops every run. Nothing for samples that are old or missing. ### `alert_html` ```mojo def alert_html(mut h: Html, w: List[String]) ``` The warnings as a box linking to `/ops/control`, or nothing. ### `warnings_json` ```mojo def warnings_json(w: List[String]) -> String ``` ### `range_hours` ```mojo def range_hours(range: String) -> Int ``` The hours `?range=` asks for: 24, 168 (`7d`) or 336 (`14d`). ### `system_charts` ```mojo def system_charts(mut db: State, hours: Int) -> Charts ``` The minutes in the last `hours`, in at most 336 points: every 5 minutes for a day, 30 for a week, an hour for two weeks. A point with no samples is 0. ### `system_html` ```mojo def system_html(mut h: Html, now: Optional[Now], c: Charts, range: String) ``` The cards, the containers' table and the charts. ### `now_json` ```mojo def now_json(now: Optional[Now]) -> String ``` ### `system_json` ```mojo def system_json(now: Optional[Now], c: Charts) -> String ``` ### `prometheus` ```mojo def prometheus(now: Optional[Now], f: Summary) -> String ``` The gauges, in Prometheus' text format: the fleet's (`f`, from `orca.fleet`) and the control instance's, which are left out while its samples are stale or missing. ## Aliases ### `KEEP_DAYS` ```mojo comptime KEEP_DAYS = 14 ``` How long the minutes, the containers' and the day files are kept. ### `DISK_WARN` ```mojo comptime DISK_WARN = SIMD(85) ``` The fullest disk's percent over which the home page warns. ### `MEMORY_WARN` ```mojo comptime MEMORY_WARN = SIMD(90) ``` Memory's percent over which the home page warns. ### `FRESH_SECONDS` ```mojo comptime FRESH_SECONDS = 600 ``` How old the newest sample may be and still be what's happening now.