watchdog_recovered
def watchdog_recovered(mut db: State)Resolves the watchdog alerts whose trips were acknowledged or whose pipelines were unpaused.
orca.alerts
What the scheduler raises alerts about (orca.notify, #184), each decided where the data is and sent through orca.notify.alert:
| Kind | Subject | Priority | When |
|---|---|---|---|
watchdog |
the rule | high | a rule tripped (sent from orca.watchdog.trip); resolved, low, when no unacknowledged trip of it has a pipeline still paused |
run-failed |
run and start | default | a run failed with no retry left (failed_runs), once per run |
fill-stalled |
the pipeline | high | fill: no fill run finished in ORCA_NOTIFY_FILL_STALL_HOURS (2) while the pipeline is unpaused, inside its window and not done; resolved when one finishes |
disk |
the mount | high | disk: the control host's fullest disk is over ORCA_NOTIFY_DISK_PERCENT (85); resolved when it isn't |
worker-lost |
worker and last seen | default | lost_workers: a worker that last said it was running runs, quiet for ORCA_NOTIFY_WORKER_SECONDS (300), once per loss |
backup-failed |
nightly |
high | orca notify --kind backup-failed ..., from orca-stack's backup |
The text is plain and short: names, numbers and the first line of an error, never a secret.
Functions
failed_runs
def failed_runs(mut db: State, pipeline: String, run_retries: Int, retried_sql: String) -> IntAlerts for pipeline's runs that failed in the last hour with no retry left: past run_retries starts, or of a trigger the scheduler doesn't retry (retried_sql), or not started for their code. Once per run and start. Returns how many were raised.
fill
def fill(mut db: State, pipeline: String, s: FillStatus, paused: Bool, in_window: Bool)Raises 'fill stalled' when no fill run of pipeline finished in ORCA_NOTIFY_FILL_STALL_HOURS (2) although it is unpaused, inside its window and not done; resolves it once one finishes.
disk
def disk(mut db: State)Raises 'disk' when the control host's fullest disk, from the /system sampler, is over ORCA_NOTIFY_DISK_PERCENT (85); resolves it when not.
lost_workers
def lost_workers(mut db: State) -> IntAlerts for the workers (orca.workers) that last said they were running runs and haven't been heard from for ORCA_NOTIFY_WORKER_SECONDS (300), once per loss. Their runs are requeued when their heartbeats go stale (orca.scheduler). Returns how many were raised.