Operations

Alerts and the watchdog

Problems show in the UI first. Alerts push the important ones to your phone, and the watchdog pauses pipelines when something leaves a sane range. This page says what triggers each, where the numbers come from, and where to look when a run is stuck.

Alerts

With an ntfy topic set, the scheduler pushes alerts to the ntfy phone app, each with a link into the UI:

export ORCA_NOTIFY_URL=https://ntfy.sh/$(openssl rand -hex 16)   # subscribe to this topic in the app
export ORCA_UI_URL=https://orca.example.com                      # where the notifications link to
orca notify test                                                 # the phone shows "orca: test alert"

Anyone who knows the topic can read what is sent to it, so treat its name as a secret. On the Hetzner server, NTFY_TOPIC_FILE in hetzner.env sets this up.

What is sent

The same alert, by kind and subject, is sent at most once per ORCA_NOTIFY_MIN_SECONDS (1800). What was sent is kept in Postgres, so a restart doesn't send it again. A send that fails is logged without the URL and tried again later, with a wait that doubles from 30 seconds to 15 minutes. It never blocks or fails the scheduler.

Settings

The scheduler reads ORCA_NOTIFY_URL (a topic URL, or a server when the topic comes from a secret), ORCA_NOTIFY_SECRET (a secret holding the topic), ORCA_NOTIFY_TOKEN or ORCA_NOTIFY_TOKEN_SECRET (an access token for a protected topic) and ORCA_UI_URL (for the links). The thresholds above each have a variable of their own. None of these is logged or put in a message. orca.toml and settings lists them, and alerts.md has the full detail.

From a script

orca notify --kind backup-failed --subject nightly --priority high "pg_dump failed: $err"

orca notify sends an alert held back like the scheduler's. The Hetzner server's nightly backup uses it when the backup fails.

Ops › Alerts

Ops › Alerts (/ops/alerts, /api/alerts, orca alerts) lists every alert sent, newest first: its kind, title and message, links to the run, pipeline, worker or watchdog rule, whether ntfy took it, and whether an alert that ends is still open. Alerts are kept 90 days. Ops › Control says whether alerts are on and lists the last few.

The watchdog

The watchdog runs inside the scheduler every ORCA_WATCHDOG_SECONDS (60; 0 turns it off) and reads Postgres only. When a rule passes one of its thresholds it pauses the pipelines in its scope, records a trip, shows a red banner on /, Ops › Control and Ops › Pools, and sends a high-priority alert. It never resumes anything: a person decides when a pipeline runs again. Runs already going go on.

Rules

Each threshold has an ORCA_WATCHDOG_* variable of its own. On the local runner, as on the Hetzner server, nothing is launched, so there is nothing to price: cost reads "fixed price" and never trips. The other rules still apply. The cost elsewhere is an estimate from an on-demand price table.

After a trip

orca watchdog status             # each rule, each threshold and value, and the trips
orca watchdog snooze launches 2h # silence a rule while you fix what trips it
orca watchdog ack --all          # keep the pipelines paused, hide the banner
orca unpause gharchive           # run again; this clears the trip

Fix the cause first: a rule still past its limit trips again on the next pass. Unpausing a trip's pipelines clears it and sends a "resolved". Every pause, ack and snooze is audited. /watchdog shows the same as orca watchdog status, with a snooze form, and every command works from a laptop with orca --remote <url> (watchdog.md).

Metrics

Machine metrics

Beside each run's process, the runner starts <binary> sample, which writes a line of the machine's CPU, memory, disk, disk I/O and network every ORCA_SAMPLE_SECONDS (5) to the run's metrics/ folder under ORCA_LOGS. A remote worker does the same beside each run it starts. A run's page draws them as its Machines card, on the same time axis as its steps. They are the whole machine's numbers, shared with whatever else ran there; the run's own peak memory and CPU are in its Resources card (Machine metrics).

Ops › Control and Ops › Pools

• Ops › Control (/ops/control, /api/system) is the machine running the scheduler, the UI and Postgres: CPU, memory, disk, network and each container's memory over 24 hours, 7 or 14 days, its memory pressure and whether new runs wait for it, and the local and control-placed runs going, each with its cgroup's memory against its limit.
• Ops › Pools (/ops/pools, /api/pools) is every other machine runs are placed on: EC2 instances, Kubernetes pods and remote workers, with each one's state and last heartbeat. With EC2 in view it also has hosts, cost and runs per hour, and Hosts over time: the hosts that ran during each hour, by role (EC2 heads and workers, the control host, and, since 0.23.0, self-hosted workers that ran a run or were seen that hour).

/metrics

/metrics has Prometheus gauges for the watchdog (orca_watchdog_tripped, _value and _threshold, by rule) and for memory pressure (orca_memory_pressure_some_avg10 and the others, and orca_admission_memory_pressure_held, by host).

Where to look when a run is stuck

1. The run's page. A queued run's Status card says why it waits (held and waiting in its JSON): no room in the grants, memory pressure, max_runs, or waiting for a worker of its pool. The pipeline's page has the same under Waiting, and orca status <pipeline> prints it.
2. Paused? A watchdog trip or a deploy pauses pipelines, and a paused pipeline starts nothing. Look for the red banner, or run orca watchdog status.
3. Its machine. For a running run, the Machines card shows whether the machine is busy, out of memory or out of disk. For a pool, Ops › Pools shows whether its worker is live and when it was last heard from; a worker gone for half ORCA_STALE_SECONDS has its runs queued again.
4. The host's pressure. Ops › Control's Memory pressure card, or memory_pressure in /api/workers for a worker.
5. The logs. The run's driver log and its steps' logs, linked from its page.

For a run that failed rather than stalled, see Debugging failures.