morca

morca · API

orca.notify

Alerts to a person: a push to the owner's phone through ntfy (#184).

A problem is otherwise only visible in the UI. These are raised where they're decided (the watchdog's trip, a run failing after its last retry, a fill that stopped moving, the control host's disk, a worker that went quiet, and orca notify from a shell script such as the nightly backup) and sent as one HTTP POST of plain text to an ntfy topic URL, with the headers ntfy reads:

Title, Priority (high | default | low), Tags, Click (the UI's link)

Settings. None of these is logged, and none is ever in a message.

Variable Meaning
ORCA_NOTIFY_URL a topic URL (https://ntfy.sh/<long-random-topic>), or just a server (https://ntfy.example.com) when the topic comes from the secret. Unset, and no ORCA_NOTIFY_SECRET: no alerts.
ORCA_NOTIFY_SECRET the id of a secret, read with ORCA_SECRET_FETCH (orca.control.fetch_secret), holding the topic: a name (appended to ORCA_NOTIFY_URL, else https://ntfy.sh), or a whole URL. The topic is a secret in practice (anyone who knows it can read it).
ORCA_NOTIFY_TOKEN, ORCA_NOTIFY_TOKEN_SECRET an ntfy access token for a protected topic, plain or as a secret's id.
ORCA_NOTIFY_MIN_SECONDS 1800: the same alert (kind and subject) is sent at most once per period.
ORCA_NOTIFY_TIMEOUT_MS 5000: how long a send may take.
ORCA_NOTIFY_FILL_STALL_HOURS 2: see orca.scheduler.
ORCA_NOTIFY_DISK_PERCENT 85: the control host's fullest disk.
ORCA_NOTIFY_WORKER_SECONDS 300: a worker holding runs, quiet for this long.
ORCA_UI_URL where Click goes (<ORCA_UI_URL>/runs/<id>, /ops/control).

Dedup and retries are in Postgres (orca.notifications, a row per kind and subject), so a restart doesn't send again. alert records the alert as firing and tries to send it; one that failed stays pending and flush, which the scheduler calls on every tick, tries it again after a wait that doubles (30 seconds, up to 15 minutes). A send is one POST with a bounded timeout and no retry of its own, and a failure is logged without the URL, so a down ntfy costs the scheduler at most one timeout a tick, and never an exception. resolve sends a low-priority 'resolved' once, for an alert that was sent.

Each one is kept (orca.alerts, Ops › Alerts): a row per alert raised and per 'resolved', with its words, its link, the pipelines and run it is about, whether it was delivered (delivery: pending, sent, or dropped when a newer one replaced it before it went, with the last error of a try that failed) and, on a raised one, when it resolved (resolved_at, and the row that resolved it, or a later raise of the same alert). Held-back alerts aren't sent, so they have no row. Kept 90 days.

Structs

Target

struct Target

Where to send, and the token to send with. Never printed.

Implements Movable

Fields
url: String
token: String

Sent

struct Sent

Implements Copyable, Movable

Fields
kind: String
subject: String
state: String
title: String
priority: String
when: String
ago: Int
pending: Bool
error: String

Functions

min_seconds

def min_seconds() -> Int

ORCA_NOTIFY_MIN_SECONDS: how long the same alert is held back.

configured

def configured() -> Bool

Whether alerts are on: the URL or the secret naming the topic is set. Reads only the environment (the secret is read when sending).

ui_link

def ui_link(path: String) -> String

path on the UI at ORCA_UI_URL, or empty when that isn't set.

target

def target() -> Target

The topic URL and token from the settings, reading the secrets. The errors say which setting, never a value.

post

def post(t: Target, title: String, body: String, priority: String, tags: String, click: String)

One POST to ntfy. Raises a message that holds neither the URL nor the token: HTTP <status> for a refusal, else that it couldn't connect.

send_now

def send_now(title: String, body: String, priority: String = "default", tags: String = "", click: String = "")

Sends at once, without Postgres: orca notify test, and the fallback of orca notify when the database isn't there.

alert

def alert(mut db: State, kind: String, subject: String, priority: String, tags: String, title: String, body: String, click: String = "", pipelines: String = "", run: Int = Int(0)) -> Bool

Records the alert kind about subject as firing and sends it, unless the same one was sent in the last min_seconds() (or is pending). Returns whether it will be sent (now, or by a later flush). pipelines (comma separated) and run are what it is about, for its row on Ops › Alerts. Never raises for a send that fails; Postgres errors do.

is_firing

def is_firing(mut db: State, kind: String, subject: String) -> Bool

Whether an alert of this kind and subject was raised and not yet resolved.

firing_subjects

def firing_subjects(mut db: State, kind: String) -> List[String]

The subjects of kind raised and not yet resolved.

resolve

def resolve(mut db: State, kind: String, subject: String, title: String, body: String, click: String = "")

Ends a firing alert: a low-priority 'resolved' is sent once, if the alert itself was ever sent. Nothing for an alert that isn't firing.

flush

def flush(mut db: State) -> Int

Sends the alerts that are pending and due (a few a call), and returns how many were sent. A send that fails is logged and tried again later; it never raises. Postgres errors do.

recent

def recent(mut db: State, limit: Int = Int(8)) -> List[Sent]

The alerts last sent or waiting to be, newest first.

notify_json

def notify_json(on: Bool, sent: List[Sent]) -> String

Whether alerts are configured and the last few; never the URL.

notify_html

def notify_html(mut h: Html, on: Bool, sent: List[Sent])

Ops > Control's alerts: whether they're configured, and the last few. The topic is a secret, so only whether it is set is shown.

Aliases

comptime DEFAULT_MIN_SECONDS = 1800
comptime DEFAULT_SERVER = "https://ntfy.sh"