morca

morca · API

orca.fleet

The fleet: every machine orca is paying for, as Ops' Pools shows the EC2 ones (/ops/pools?kind=ec2, orca.ui.pools) and /api/fleet has them as JSON.

What the page shows comes from three places, none of which reads the samplers' files when a page is viewed:

  • orca.instances says which hosts exist, since when, for which run;
  • orca.fleet_samples holds each live host's last 30 minutes of samples, one row per sample;
  • orca.fleet_hourly holds, per hour and role, what the 7-day charts add up (CPU seconds used and available, bytes in and out), kept 8 days.

The scheduler fills both tables (ingest, every ORCA_FLEET_SECONDS, 30; 0 turns it off). Run instances upload their samples to the run's metrics/ folder (orca.sample) every ORCA_CLUSTER_LOG_SECONDS, so ingest reads the last lines of each live instance's file from there, a few hundred bytes of it per instance per pass in effect, and keeps those it hasn't got. The control instance's sampler (orca sample --out <file> --keep N, started by the stack's user data) writes a file ORCA_CONTROL_SAMPLES names, and ORCA_CONTROL_TYPE says what the instance is. Without them the control instance has no row, unless runs placed on it (orca register --placement control, orca.control) are running, which its row lists; without ORCA_LOGS no instance has one.

Prices are a small table of us-east-2 on-demand rates for the types orca runs on: an estimate (Spot hosts cost less, and a stopped-and-started clock isn't modelled). With ORCA_RUNNER=local no instance is launched, so the cost tile says "fixed price" instead and shows ORCA_FIXED_COST (free text, such as €8.49/month; "Fixed" when unset), with no region.

Structs

FleetSample

struct FleetSample

What the fleet keeps of a sampler's line.

Implements Copyable, Movable

Fields
t: Float64: Unix time.
dt: Float64
cores: Float64
cpu: Float64: User and system, percent of all cores.
busiest: Float64
mem: Float64: Percent of memory used.
disk: Float64: Percent used of the fullest disk.
net_in: Float64: Bytes per second.
net_out: Float64

Host

struct Host

Implements Copyable, Movable

Fields
id: String
role: String: head, worker or control.
instance_type: String
run: Int: The run it serves; 0 for the control instance.
uptime: Float64
state: String: starting, running, draining, idle (pooled) (its run ended, and it waits in the EC2 runner's warm pool for another) or control.
sampled: Bool
age: Float64: Seconds since its last sample.
cpu: Float64
busiest: Float64
mem: Float64
disk: Float64
net_in: Float64
net_out: Float64
spark_x: List[Float64]: Its CPU over the last 10 minutes, as (0 = then, 1 = now) and percent.
spark_y: List[Float64]
idle: Float64: Seconds it has been idle in the pool; 0 when it isn't.
runs: List[Int]: The runs it serves: its run, or for the control instance the control-placed runs running there (orca register --placement control).

TypeCount

struct TypeCount

Implements Copyable, Movable

Fields
instance_type: String
n: Int

Summary

struct Summary

Implements Copyable, Movable

Fields
heads: Int
workers: Int
control: Bool
types: List[TypeCount]
cost: Float64: US dollars an hour at on-demand prices.
unpriced: Int: Hosts whose type isn't in the price table.
running: Int
queued: Int
runs_hour: Int: Runs started in the last 60 minutes.
runs_spark: List[Float64]: Per hour over the last 24, the last one the current hour.
failed_attempts_hour: Int
failed_runs_hour: Int
errors_spark: List[Float64]: Failed attempts and failed runs per hour.

hosts

def hosts(self) -> Int

Series

struct Series

Implements Copyable, Movable

Fields
name: String
cls: String
values: List[Float64]
over: Bool: Drawn over the series before it, as a bar of the same place.
area: Bool

__init__

def __init__(out self, name: String, cls: String, var values: List[Float64], over: Bool = False, area: Bool = False)

Charts

struct Charts

Implements Copyable, Movable

Fields
hours: Int
labels: List[String]: MM-DD HH:00 (UTC) of each hour.
hosts: List[Series]
runs: List[Series]
cpu: List[Series]
net: List[Series]

Functions

hourly_price

def hourly_price(instance_type: String) -> Float64

The us-east-2 on-demand price per hour in US dollars of an instance type orca runs on, or -1 for any other.

parse_fleet_sample

def parse_fleet_sample(line: String) -> FleetSample

A sampler's line; raises if it isn't one.

tail_samples

def tail_samples(text: String, n: Int = Int(120)) -> List[FleetSample]

The samples among the last n lines of a sampler's file, oldest first; lines that aren't samples are skipped.

ingest

def ingest(mut db: State) -> Int

One pass: the new samples of every live instance (and of the control instance), kept; old ones dropped. Returns the samples added. An instance whose file can't be read is skipped, since it may not have uploaded yet.

fleet_hosts

def fleet_hosts(mut db: State) -> List[Host]

One host per live instance, with its last sample, then the control instance if it is sampled.

fleet_summary

def fleet_summary(mut db: State) -> Summary

fleet_charts

def fleet_charts(mut db: State, hours: Int) -> Charts

sparkline

def sparkline(xs: List[Float64], ys: List[Float64], top: Float64, label: String) -> String

A small line of ys (0 to top) at xs (0 to 1) as an inline SVG; an empty box without points.

summary_html

def summary_html(mut h: Html, s: Summary, compact: Bool)

The summary cards; each links to where its detail is.

sort_hosts

def sort_hosts(mut hosts: List[Host], sort: String, desc: Bool)

Orders the hosts by a column, as ?sort= and &dir=desc say; an unknown column is the default (role, then id).

charts_html

def charts_html(mut h: Html, c: Charts, range: String, base: String)

The cards of charts, and the choice of range, links to base (a path and maybe a query) with range= added.

summary_json

def summary_json(s: Summary) -> String

hosts_json

def hosts_json(hosts: List[Host]) -> String

charts_json

def charts_json(c: Charts) -> String

Aliases

comptime SAMPLE_KEEP_MINUTES = 30

How long a host's samples are kept in orca.fleet_samples.

comptime HOURLY_KEEP_DAYS = 8