# orca.watchdog

> A module of morca's API.

The watchdog: pauses pipelines when something leaves a sane range.

On 2026-11-05 nothing noticed ~360 EC2 launches in 38 minutes, or a run
waiting three hours for workers that never came. The scheduler evaluates
these rules every `ORCA_WATCHDOG_SECONDS` (60; 0 turns the watchdog off),
each from Postgres only (never object storage), with the thresholds below
(each rule's environment variable overrides its default):

| Rule | Trips when | Scope |
|---|---|---|
| `launches` | more than `ORCA_WATCHDOG_LAUNCHES` (200) instances were launched in the last hour (pool handovers aren't launches) | all pipelines |
| `cost` | the estimated cost of the instances up (`orca.fleet`'s price table) stayed over `ORCA_WATCHDOG_COST_HOUR` ($3) an hour for `ORCA_WATCHDOG_COST_MINUTES` (15) minutes, or the last 24 hours cost over `ORCA_WATCHDOG_COST_DAY` ($25) | all pipelines |
| `idle` | at least `ORCA_WATCHDOG_IDLE_WORKERS` (4) workers, up for `ORCA_WATCHDOG_IDLE_MINUTES` (10, at most 30) minutes, averaged under `ORCA_WATCHDOG_IDLE_CPU` (15) percent CPU over them (pooled idle and draining workers aren't counted) | the pipelines owning those workers |
| `failing` | a pipeline had more than `ORCA_WATCHDOG_FAILED` (20) failed attempts in the last hour, or at least `ORCA_WATCHDOG_FAILED_MIN` (10) attempts ended and more than `ORCA_WATCHDOG_FAILED_PERCENT` (50) percent failed | each such pipeline |
| `stuck` | a run with a running attempt, on instances that are sampled, has had no attempt start or end, no instance launched or ended and no sampled host over `ORCA_WATCHDOG_STUCK_CPU` (5) percent CPU for `ORCA_WATCHDOG_STUCK_MINUTES` (30, at most 30, since samples are kept that long) minutes | that run's pipeline |
| `control` | the control instance's newest minute has memory over `ORCA_WATCHDOG_MEMORY` (90) percent or its fullest disk over `ORCA_WATCHDOG_DISK` (85) | all pipelines |

A tripped rule **pauses** its scope through `orca.pause.set_paused`
(what the UI's Pause button does; pipelines already paused are left
alone), writes an `orca.audit` row per pipeline (user `watchdog`, the
rule's numbers), and a row of `orca.watchdog_trips`, which the banner on
`/`, `/ops/pools` and `/ops/control`, `/watchdog` and `/metrics` show for as long as any of its
pipelines is paused. It never resumes anything. A rule that is still over
its limit when a pipeline is unpaused trips again on the next pass, so
snooze it (`orca watchdog snooze <rule> 2h`, or on `/watchdog`) first;
that is stored in `orca.watchdog_rules` and audited.

A rule checks one threshold or more (`limits`: `control` checks memory
and disk, `cost` the sustained rate and the day, `failing` the count and
the share, `idle` the CPU and the number of workers). Each reading keeps
every check's value and limit and whether it is past it, and so does each
trip, so `orca watchdog status`, `/watchdog` and `/api/watchdog` show
every threshold and which one tripped.

Evaluation is pure (`eval_*`, over plain rows) apart from the gathering
(`gather_*`).

## Structs

### `Config`

```mojo
struct Config
```

Implements: `Copyable`, `Movable`

Fields:

- `seconds: Float64`
- `launches: Int`
- `cost_hour: Float64`
- `cost_minutes: Int`
- `cost_day: Float64`
- `idle_cpu: Float64`
- `idle_minutes: Int`
- `idle_workers: Int`
- `failed: Int`
- `failed_percent: Float64`
- `failed_min: Int`
- `stuck_minutes: Int`
- `stuck_cpu: Float64`
- `memory: Float64`
- `disk: Float64`
- `control_price: Float64`: Dollars an hour of the control instance (`ORCA_CONTROL_TYPE`), or 0.
- `fixed_price: Bool`: The runner launches no instances (`ORCA_RUNNER=local`), so the cost rule has nothing to price: it reads "fixed price" and never trips.

### `Check`

```mojo
struct Check
```

One threshold a rule checks, and what it saw against it.

Implements: `Copyable`, `Movable`

Fields:

- `name: String`
- `value: Float64`: -1 when there is nothing to judge.
- `limit: Float64`
- `unit: String`
- `op: String`: How the value trips it: `over`, `under` or `at least` the limit.
- `past: Bool`: The value is past the limit (the rule trips when all of its conditions are; for `idle` both checks must be past).

### `Reading`

```mojo
struct Reading
```

What a rule saw: its current value against its threshold, and if that is a trip, why and which pipelines.

Implements: `Copyable`, `Movable`

Fields:

- `rule: String`
- `value: Float64`: -1 when there is nothing to judge (no samples).
- `threshold: Float64`: The limit of the check nearest it (`value` is that check's).
- `tripped: Bool`
- `detail: String`
- `pipelines: List[String]`
- `checks: List[Check]`: Every threshold the rule checks (`limits`), with its value.

### `Lease`

```mojo
struct Lease
```

An instance's row: its type, how long ago it started and ended (-1: it hasn't).

Implements: `Copyable`, `Movable`

Fields:

- `instance_type: String`
- `start_ago: Float64`
- `end_ago: Float64`

### `Counts`

```mojo
struct Counts
```

A pipeline's attempts that ended in the last hour.

Implements: `Copyable`, `Movable`

Fields:

- `pipeline: String`
- `failed: Int`
- `total: Int`

### `Quiet`

```mojo
struct Quiet
```

A run with a running attempt, and the minutes nothing has moved.

Implements: `Copyable`, `Movable`

Fields:

- `pipeline: String`
- `run: Int`
- `minutes: Float64`

### `Trip`

```mojo
struct Trip
```

Implements: `Copyable`, `Movable`

Fields:

- `rule: String`
- `ago: Float64`
- `at: String`
- `value: Float64`
- `threshold: Float64`
- `detail: String`
- `pipelines: List[String]`: The pipelines it paused that are still paused.
- `id: Int`
- `acked_by: String`: Who kept it paused (acknowledged it), or empty.
- `checks: List[Check]`: The checks as the trip saw them (empty for a trip from before they were kept).

### `RuleRow`

```mojo
struct RuleRow
```

Implements: `Copyable`, `Movable`

Fields:

- `rule: String`
- `value: Float64`
- `threshold: Float64`
- `checked: Bool`
- `snoozed_until: String`: UTC, or empty.
- `snoozed_by: String`
- `tripped: Bool`
- `note: String`: Why the rule has no reading ("fixed price"), or empty.
- `checks: List[Check]`: Each threshold with its last value (-1 before the first check).
- `tripped_on: List[String]`: The checks its unacknowledged trips tripped on (what `tripped` shows).

## Functions

### `rule_names`

```mojo
def rule_names() -> List[String]
```

### `is_rule`

```mojo
def is_rule(name: String) -> Bool
```

### `default_config`

```mojo
def default_config() -> Config
```

### `config`

```mojo
def config() -> Config
```

The thresholds, from the environment.

### `limits`

```mojo
def limits(c: Config, rule: String) -> List[Check]
```

The thresholds `rule` checks, with no values yet.

### `checks_json`

```mojo
def checks_json(checks: List[Check]) -> String
```

### `parse_checks`

```mojo
def parse_checks(text: String) -> List[Check]
```

`checks_json`'s text back (empty: none).

### `past_names`

```mojo
def past_names(checks: List[Check]) -> List[String]
```

### `eval_launches`

```mojo
def eval_launches(c: Config, launched: Int, everything: List[String]) -> Reading
```

`launched` instances in the last hour against the limit.

### `cost_rate`

```mojo
def cost_rate(leases: List[Lease], ago: Float64, control: Float64) -> Float64
```

Dollars an hour being spent `ago` seconds ago (unpriced types cost nothing here, as in `orca.fleet`).

### `cost_spent`

```mojo
def cost_spent(leases: List[Lease], window: Float64, control: Float64) -> Float64
```

Dollars spent over the last `window` seconds.

### `eval_cost`

```mojo
def eval_cost(c: Config, leases: List[Lease], everything: List[String]) -> Reading
```

The rate now (the reading's value), the lowest it was at each minute of the sustain period, and the last 24 hours' spend. On the local runner (`fixed_price`) there is nothing to price: no reading, no trip.

### `eval_idle`

```mojo
def eval_idle(c: Config, workers: Int, cpu: Float64, pipelines: List[String]) -> Reading
```

`workers` up for the whole window, averaging `cpu` percent over it (-1: no samples).

### `eval_failing`

```mojo
def eval_failing(c: Config, counts: List[Counts]) -> Reading
```

Pipelines over either failure limit; the value is the most failed attempts any had; the share check's, the highest share of a pipeline with at least `failed_min` attempts ended.

### `eval_stuck`

```mojo
def eval_stuck(c: Config, runs: List[Quiet]) -> Reading
```

### `eval_control`

```mojo
def eval_control(c: Config, have: Bool, mem: Float64, disk: Float64, everything: List[String]) -> Reading
```

The newest minute's memory and fullest disk (`have` false: no fresh samples). The value is whichever is nearer its limit, in percent; the checks have both.

### `all_pipelines`

```mojo
def all_pipelines(mut db: State) -> List[String]
```

### `gather_launches`

```mojo
def gather_launches(mut db: State) -> Int
```

### `gather_leases`

```mojo
def gather_leases(mut db: State) -> List[Lease]
```

### `gather_idle`

```mojo
def gather_idle(mut db: State, c: Config) -> Tuple[Int, Float64, List[String]]
```

The workers up the whole window that were sampled, their average CPU (-1 if none), and the pipelines that own them.

### `gather_failing`

```mojo
def gather_failing(mut db: State) -> List[Counts]
```

### `gather_stuck`

```mojo
def gather_stuck(mut db: State, c: Config) -> List[Quiet]
```

### `evaluate`

```mojo
def evaluate(mut db: State, c: Config) -> List[Reading]
```

Every rule's reading now.

### `snoozed`

```mojo
def snoozed(mut db: State, rule: String) -> Bool
```

### `trip`

```mojo
def trip(mut db: State, r: Reading) -> List[String]
```

Pauses the unpaused pipelines in `r`'s scope and records why. Returns the ones it paused; none, and no record, if all were paused already.

### `check`

```mojo
def check(mut db: State, c: Config) -> Int
```

One pass: evaluates the rules, keeps their readings, and trips those over their limits that aren't snoozed. Returns the pipelines paused.

### `parse_duration`

```mojo
def parse_duration(text: String) -> Int
```

`90s`, `30m`, `2h` or `1d` in seconds; `off` is 0 (end a snooze).

### `snooze`

```mojo
def snooze(mut db: State, rule: String, duration: String, by: String) -> Int
```

Silences `rule` for `duration` (`off` ends it). Returns the seconds.

### `active_trips`

```mojo
def active_trips(mut db: State) -> List[Trip]
```

The trips with a pipeline still paused that nobody has acknowledged, newest first: what the banner shows.

### `listed_trips`

```mojo
def listed_trips(mut db: State) -> List[Trip]
```

Like `active_trips`, with the acknowledged ones: what status lists.

### `acknowledge`

```mojo
def acknowledge(mut db: State, trip: Int, by: String) -> Int
```

Keeps the pipelines of the unacknowledged trips (all of them if `trip` is 0) paused, and stops the banner showing them. Returns how many.

### `rule_rows`

```mojo
def rule_rows(mut db: State, c: Config) -> List[RuleRow]
```

Each rule's last reading and snooze; thresholds from `c` where the scheduler hasn't read it yet.

### `unit_of`

```mojo
def unit_of(rule: String) -> String
```

### `banner_html`

```mojo
def banner_html(mut h: Html, trips: List[Trip])
```

A red box listing the trips with a pipeline still paused, or nothing.

### `check_text`

```mojo
def check_text(k: Check, digits: Int = Int(2)) -> String
```

`memory 10.60 % (trips over 90.00)`, and `, past` when it is.

### `trips_json`

```mojo
def trips_json(trips: List[Trip]) -> String
```

### `rules_json`

```mojo
def rules_json(rows: List[RuleRow]) -> String
```

### `watchdog_json`

```mojo
def watchdog_json(mut db: State, c: Config) -> String
```

### `watchdog_html`

```mojo
def watchdog_html(mut h: Html, rows: List[RuleRow], trips: List[Trip])
```

The `/watchdog` page's body: the rules, with a snooze form each.

### `prometheus`

```mojo
def prometheus(mut db: State, c: Config) -> String
```

`orca_watchdog_tripped{rule}` (1 while a trip of the rule has a pipeline paused), and the rule's last value and threshold.

### `status_lines`

```mojo
def status_lines(json: String) -> List[String]
```

`orca watchdog status`'s lines from `/api/watchdog`'s JSON.

## Aliases

### `SAMPLE_MINUTES`

```mojo
comptime SAMPLE_MINUTES = 30
```

How long fleet samples are kept, which bounds `idle` and `stuck`.

### `MAX_SNOOZE_SECONDS`

```mojo
comptime MAX_SNOOZE_SECONDS = 2592000
```