morca

morca · API

orca.watchdog

The watchdog: pauses pipelines when something leaves a sane range.

On 2026-11-05 nothing noticed ~360 EC2 launches in 38 minutes, or a run waiting three hours for workers that never came. The scheduler evaluates these rules every ORCA_WATCHDOG_SECONDS (60; 0 turns the watchdog off), each from Postgres only (never object storage), with the thresholds below (each rule's environment variable overrides its default):

Rule Trips when Scope
launches more than ORCA_WATCHDOG_LAUNCHES (200) instances were launched in the last hour (pool handovers aren't launches) all pipelines
cost the estimated cost of the instances up (orca.fleet's price table) stayed over ORCA_WATCHDOG_COST_HOUR ($3) an hour for ORCA_WATCHDOG_COST_MINUTES (15) minutes, or the last 24 hours cost over ORCA_WATCHDOG_COST_DAY ($25) all pipelines
idle at least ORCA_WATCHDOG_IDLE_WORKERS (4) workers, up for ORCA_WATCHDOG_IDLE_MINUTES (10, at most 30) minutes, averaged under ORCA_WATCHDOG_IDLE_CPU (15) percent CPU over them (pooled idle and draining workers aren't counted) the pipelines owning those workers
failing a pipeline had more than ORCA_WATCHDOG_FAILED (20) failed attempts in the last hour, or at least ORCA_WATCHDOG_FAILED_MIN (10) attempts ended and more than ORCA_WATCHDOG_FAILED_PERCENT (50) percent failed each such pipeline
stuck a run with a running attempt, on instances that are sampled, has had no attempt start or end, no instance launched or ended and no sampled host over ORCA_WATCHDOG_STUCK_CPU (5) percent CPU for ORCA_WATCHDOG_STUCK_MINUTES (30, at most 30, since samples are kept that long) minutes that run's pipeline
control the control instance's newest minute has memory over ORCA_WATCHDOG_MEMORY (90) percent or its fullest disk over ORCA_WATCHDOG_DISK (85) all pipelines

A tripped rule pauses its scope through orca.pause.set_paused (what the UI's Pause button does; pipelines already paused are left alone), writes an orca.audit row per pipeline (user watchdog, the rule's numbers), and a row of orca.watchdog_trips, which the banner on /, /ops/pools and /ops/control, /watchdog and /metrics show for as long as any of its pipelines is paused. It never resumes anything. A rule that is still over its limit when a pipeline is unpaused trips again on the next pass, so snooze it (orca watchdog snooze <rule> 2h, or on /watchdog) first; that is stored in orca.watchdog_rules and audited.

A rule checks one threshold or more (limits: control checks memory and disk, cost the sustained rate and the day, failing the count and the share, idle the CPU and the number of workers). Each reading keeps every check's value and limit and whether it is past it, and so does each trip, so orca watchdog status, /watchdog and /api/watchdog show every threshold and which one tripped.

Evaluation is pure (eval_*, over plain rows) apart from the gathering (gather_*).

Structs

Config

struct Config

Implements Copyable, Movable

Fields
seconds: Float64
launches: Int
cost_hour: Float64
cost_minutes: Int
cost_day: Float64
idle_cpu: Float64
idle_minutes: Int
idle_workers: Int
failed: Int
failed_percent: Float64
failed_min: Int
stuck_minutes: Int
stuck_cpu: Float64
memory: Float64
disk: Float64
control_price: Float64: Dollars an hour of the control instance (ORCA_CONTROL_TYPE), or 0.
fixed_price: Bool: The runner launches no instances (ORCA_RUNNER=local), so the cost rule has nothing to price: it reads "fixed price" and never trips.

Check

struct Check

One threshold a rule checks, and what it saw against it.

Implements Copyable, Movable

Fields
name: String
value: Float64: -1 when there is nothing to judge.
limit: Float64
unit: String
op: String: How the value trips it: over, under or at least the limit.
past: Bool: The value is past the limit (the rule trips when all of its conditions are; for idle both checks must be past).

Reading

struct Reading

What a rule saw: its current value against its threshold, and if that is a trip, why and which pipelines.

Implements Copyable, Movable

Fields
rule: String
value: Float64: -1 when there is nothing to judge (no samples).
threshold: Float64: The limit of the check nearest it (value is that check's).
tripped: Bool
detail: String
pipelines: List[String]
checks: List[Check]: Every threshold the rule checks (limits), with its value.

Lease

struct Lease

An instance's row: its type, how long ago it started and ended (-1: it hasn't).

Implements Copyable, Movable

Fields
instance_type: String
start_ago: Float64
end_ago: Float64

Counts

struct Counts

A pipeline's attempts that ended in the last hour.

Implements Copyable, Movable

Fields
pipeline: String
failed: Int
total: Int

Quiet

struct Quiet

A run with a running attempt, and the minutes nothing has moved.

Implements Copyable, Movable

Fields
pipeline: String
run: Int
minutes: Float64

Trip

struct Trip

Implements Copyable, Movable

Fields
rule: String
ago: Float64
at: String
value: Float64
threshold: Float64
detail: String
pipelines: List[String]: The pipelines it paused that are still paused.
id: Int
acked_by: String: Who kept it paused (acknowledged it), or empty.
checks: List[Check]: The checks as the trip saw them (empty for a trip from before they were kept).

RuleRow

struct RuleRow

Implements Copyable, Movable

Fields
rule: String
value: Float64
threshold: Float64
checked: Bool
snoozed_until: String: UTC, or empty.
snoozed_by: String
tripped: Bool
note: String: Why the rule has no reading ("fixed price"), or empty.
checks: List[Check]: Each threshold with its last value (-1 before the first check).
tripped_on: List[String]: The checks its unacknowledged trips tripped on (what tripped shows).

Functions

rule_names

def rule_names() -> List[String]

is_rule

def is_rule(name: String) -> Bool

default_config

def default_config() -> Config

config

def config() -> Config

The thresholds, from the environment.

limits

def limits(c: Config, rule: String) -> List[Check]

The thresholds rule checks, with no values yet.

checks_json

def checks_json(checks: List[Check]) -> String

parse_checks

def parse_checks(text: String) -> List[Check]

checks_json's text back (empty: none).

past_names

def past_names(checks: List[Check]) -> List[String]

eval_launches

def eval_launches(c: Config, launched: Int, everything: List[String]) -> Reading

launched instances in the last hour against the limit.

cost_rate

def cost_rate(leases: List[Lease], ago: Float64, control: Float64) -> Float64

Dollars an hour being spent ago seconds ago (unpriced types cost nothing here, as in orca.fleet).

cost_spent

def cost_spent(leases: List[Lease], window: Float64, control: Float64) -> Float64

Dollars spent over the last window seconds.

eval_cost

def eval_cost(c: Config, leases: List[Lease], everything: List[String]) -> Reading

The rate now (the reading's value), the lowest it was at each minute of the sustain period, and the last 24 hours' spend. On the local runner (fixed_price) there is nothing to price: no reading, no trip.

eval_idle

def eval_idle(c: Config, workers: Int, cpu: Float64, pipelines: List[String]) -> Reading

workers up for the whole window, averaging cpu percent over it (-1: no samples).

eval_failing

def eval_failing(c: Config, counts: List[Counts]) -> Reading

Pipelines over either failure limit; the value is the most failed attempts any had; the share check's, the highest share of a pipeline with at least failed_min attempts ended.

eval_stuck

def eval_stuck(c: Config, runs: List[Quiet]) -> Reading

eval_control

def eval_control(c: Config, have: Bool, mem: Float64, disk: Float64, everything: List[String]) -> Reading

The newest minute's memory and fullest disk (have false: no fresh samples). The value is whichever is nearer its limit, in percent; the checks have both.

all_pipelines

def all_pipelines(mut db: State) -> List[String]

gather_launches

def gather_launches(mut db: State) -> Int

gather_leases

def gather_leases(mut db: State) -> List[Lease]

gather_idle

def gather_idle(mut db: State, c: Config) -> Tuple[Int, Float64, List[String]]

The workers up the whole window that were sampled, their average CPU (-1 if none), and the pipelines that own them.

gather_failing

def gather_failing(mut db: State) -> List[Counts]

gather_stuck

def gather_stuck(mut db: State, c: Config) -> List[Quiet]

evaluate

def evaluate(mut db: State, c: Config) -> List[Reading]

Every rule's reading now.

snoozed

def snoozed(mut db: State, rule: String) -> Bool

trip

def trip(mut db: State, r: Reading) -> List[String]

Pauses the unpaused pipelines in r's scope and records why. Returns the ones it paused; none, and no record, if all were paused already.

check

def check(mut db: State, c: Config) -> Int

One pass: evaluates the rules, keeps their readings, and trips those over their limits that aren't snoozed. Returns the pipelines paused.

parse_duration

def parse_duration(text: String) -> Int

90s, 30m, 2h or 1d in seconds; off is 0 (end a snooze).

snooze

def snooze(mut db: State, rule: String, duration: String, by: String) -> Int

Silences rule for duration (off ends it). Returns the seconds.

active_trips

def active_trips(mut db: State) -> List[Trip]

The trips with a pipeline still paused that nobody has acknowledged, newest first: what the banner shows.

listed_trips

def listed_trips(mut db: State) -> List[Trip]

Like active_trips, with the acknowledged ones: what status lists.

acknowledge

def acknowledge(mut db: State, trip: Int, by: String) -> Int

Keeps the pipelines of the unacknowledged trips (all of them if trip is 0) paused, and stops the banner showing them. Returns how many.

rule_rows

def rule_rows(mut db: State, c: Config) -> List[RuleRow]

Each rule's last reading and snooze; thresholds from c where the scheduler hasn't read it yet.

unit_of

def unit_of(rule: String) -> String

banner_html

def banner_html(mut h: Html, trips: List[Trip])

A red box listing the trips with a pipeline still paused, or nothing.

check_text

def check_text(k: Check, digits: Int = Int(2)) -> String

memory 10.60 % (trips over 90.00), and , past when it is.

trips_json

def trips_json(trips: List[Trip]) -> String

rules_json

def rules_json(rows: List[RuleRow]) -> String

watchdog_json

def watchdog_json(mut db: State, c: Config) -> String

watchdog_html

def watchdog_html(mut h: Html, rows: List[RuleRow], trips: List[Trip])

The /watchdog page's body: the rules, with a snooze form each.

prometheus

def prometheus(mut db: State, c: Config) -> String

orca_watchdog_tripped{rule} (1 while a trip of the rule has a pipeline paused), and the rule's last value and threshold.

status_lines

def status_lines(json: String) -> List[String]

orca watchdog status's lines from /api/watchdog's JSON.

Aliases

comptime SAMPLE_MINUTES = 30

How long fleet samples are kept, which bounds idle and stuck.

comptime MAX_SNOOZE_SECONDS = 2592000