# orca.alerts

> A module of morca's API.

What the scheduler raises alerts about (`orca.notify`, #184), each decided where the data is and sent through `orca.notify.alert`:

| Kind | Subject | Priority | When |
|---|---|---|---|
| `watchdog` | the rule | high | a rule tripped (sent from `orca.watchdog.trip`); resolved, low, when no unacknowledged trip of it has a pipeline still paused |
| `run-failed` | run and start | default | a run failed with no retry left (`failed_runs`), once per run |
| `fill-stalled` | the pipeline | high | `fill`: no fill run finished in `ORCA_NOTIFY_FILL_STALL_HOURS` (2) while the pipeline is unpaused, inside its window and not done; resolved when one finishes |
| `disk` | the mount | high | `disk`: the control host's fullest disk is over `ORCA_NOTIFY_DISK_PERCENT` (85); resolved when it isn't |
| `worker-lost` | worker and last seen | default | `lost_workers`: a worker that last said it was running runs, quiet for `ORCA_NOTIFY_WORKER_SECONDS` (300), once per loss |
| `backup-failed` | `nightly` | high | `orca notify --kind backup-failed ...`, from orca-stack's backup |

The text is plain and short: names, numbers and the first line of an error,
never a secret.

## Functions

### `watchdog_recovered`

```mojo
def watchdog_recovered(mut db: State)
```

Resolves the watchdog alerts whose trips were acknowledged or whose pipelines were unpaused.

### `failed_runs`

```mojo
def failed_runs(mut db: State, pipeline: String, run_retries: Int, retried_sql: String) -> Int
```

Alerts for `pipeline`'s runs that failed in the last hour with no retry left: past `run_retries` starts, or of a trigger the scheduler doesn't retry (`retried_sql`), or not started for their code. Once per run and start. Returns how many were raised.

### `fill`

```mojo
def fill(mut db: State, pipeline: String, s: FillStatus, paused: Bool, in_window: Bool)
```

Raises 'fill stalled' when no fill run of `pipeline` finished in `ORCA_NOTIFY_FILL_STALL_HOURS` (2) although it is unpaused, inside its window and not done; resolves it once one finishes.

### `disk`

```mojo
def disk(mut db: State)
```

Raises 'disk' when the control host's fullest disk, from the /system sampler, is over `ORCA_NOTIFY_DISK_PERCENT` (85); resolves it when not.

### `lost_workers`

```mojo
def lost_workers(mut db: State) -> Int
```

Alerts for the workers (`orca.workers`) that last said they were running runs and haven't been heard from for `ORCA_NOTIFY_WORKER_SECONDS` (300), once per loss. Their runs are requeued when their heartbeats go stale (`orca.scheduler`). Returns how many were raised.