# Alerts and the watchdog

> Watching a morca stack: the alerts it pushes to your phone through ntfy and how to set them up, the watchdog that pauses pipelines when something leaves a sane range, the machine metrics it samples, Ops › Alerts, Control and Pools, the Prometheus gauges, and where to look first when a run seems stuck.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

Problems show in the UI first. Alerts push the important ones to your phone, and the watchdog pauses pipelines when something leaves a sane range. This page says what triggers each, where the numbers come from, and where to look when a run is stuck.

## Alerts

With an [ntfy](https://ntfy.sh) topic set, the scheduler pushes alerts to the ntfy phone app, each with a link into the UI:

```
export ORCA_NOTIFY_URL=https://ntfy.sh/$(openssl rand -hex 16)   # subscribe to this topic in the app
export ORCA_UI_URL=https://orca.example.com                      # where the notifications link to
orca notify test                                                 # the phone shows "orca: test alert"
```

Anyone who knows the topic can read what is sent to it, so treat its name as a secret. On the Hetzner server, `NTFY_TOPIC_FILE` in `hetzner.env` sets this up.

### What is sent

| Kind | When | Priority |
| --- | --- | --- |
| watchdog | a watchdog rule tripped and paused pipelines | high; a low "resolved" once acknowledged or unpaused |
| run-failed | a run failed after its last retry | default |
| fill-stalled | a fill has had no fill run finish in ORCA\_NOTIFY\_FILL\_STALL\_HOURS (2) | high; "moving again" once it is |
| disk | the control host's fullest disk is over ORCA\_NOTIFY\_DISK\_PERCENT (85) | high; "fine again" once it is under |
| worker-lost | a remote worker holding runs has been quiet for ORCA\_NOTIFY\_WORKER\_SECONDS (300) | default |
| memory-pressure | a host has held new runs for memory pressure for the grace (600 s) with none of morca's runs going | high; "has eased" once it doesn't hold |
| your own | a script ran orca notify --kind <kind> | as given |

The same alert, by kind and subject, is sent at most once per `ORCA_NOTIFY_MIN_SECONDS` (1800). What was sent is kept in Postgres, so a restart doesn't send it again. A send that fails is logged without the URL and tried again later, with a wait that doubles from 30 seconds to 15 minutes. It never blocks or fails the scheduler.

### Settings

The scheduler reads `ORCA_NOTIFY_URL` (a topic URL, or a server when the topic comes from a secret), `ORCA_NOTIFY_SECRET` (a secret holding the topic), `ORCA_NOTIFY_TOKEN` or `ORCA_NOTIFY_TOKEN_SECRET` (an access token for a protected topic) and `ORCA_UI_URL` (for the links). The thresholds above each have a variable of their own. None of these is logged or put in a message. [orca.toml and settings](https://morca.dev/docs/config) lists them, and [alerts.md](https://github.com/morca-dev/morca/blob/main/docs/alerts.md) has the full detail.

### From a script

```
orca notify --kind backup-failed --subject nightly --priority high "pg_dump failed: $err"
```

`orca notify` sends an alert held back like the scheduler's. The Hetzner server's nightly backup uses it when the backup fails.

### Ops › Alerts

Ops › Alerts (`/ops/alerts`, `/api/alerts`, `orca alerts`) lists every alert sent, newest first: its kind, title and message, links to the run, pipeline, worker or watchdog rule, whether ntfy took it, and whether an alert that ends is still open. Alerts are kept 90 days. Ops › Control says whether alerts are on and lists the last few.

## The watchdog

The watchdog runs inside the scheduler every `ORCA_WATCHDOG_SECONDS` (60; 0 turns it off) and reads Postgres only. When a rule passes one of its thresholds it pauses the pipelines in its scope, records a trip, shows a red banner on `/`, Ops › Control and Ops › Pools, and sends a high-priority alert. It never resumes anything: a person decides when a pipeline runs again. Runs already going go on.

### Rules

| Rule | Trips when (defaults) | Pauses |
| --- | --- | --- |
| launches | instances launched in the last hour over 200 | all pipelines |
| cost | estimated cost over $3 an hour for 15 minutes, or over $25 in 24 hours | all pipelines |
| idle | at least 4 workers averaging under 15% CPU for 10 minutes | the pipelines owning them |
| failing | a pipeline's failed attempts in the last hour over 20, or over 50% once 10 have ended | each such pipeline |
| stuck | a running run on sampled instances with no progress and no busy host for 30 minutes | that run's pipeline |
| control | the control host's memory over 90%, or its fullest disk over 85% | all pipelines |

Each threshold has an `ORCA_WATCHDOG_*` variable of its own. On the local runner, as on the Hetzner server, nothing is launched, so there is nothing to price: `cost` reads "fixed price" and never trips. The other rules still apply. The cost elsewhere is an estimate from an on-demand price table.

### After a trip

```
orca watchdog status             # each rule, each threshold and value, and the trips
orca watchdog snooze launches 2h # silence a rule while you fix what trips it
orca watchdog ack --all          # keep the pipelines paused, hide the banner
orca unpause gharchive           # run again; this clears the trip
```

Fix the cause first: a rule still past its limit trips again on the next pass. Unpausing a trip's pipelines clears it and sends a "resolved". Every pause, ack and snooze is audited. `/watchdog` shows the same as `orca watchdog status`, with a snooze form, and every command works from a laptop with `orca --remote <url>` ([watchdog.md](https://github.com/morca-dev/morca/blob/main/docs/watchdog.md)).

## Metrics

### Machine metrics

Beside each run's process, the runner starts `<binary> sample`, which writes a line of the machine's CPU, memory, disk, disk I/O and network every `ORCA_SAMPLE_SECONDS` (5) to the run's `metrics/` folder under `ORCA_LOGS`. A remote worker does the same beside each run it starts. A run's page draws them as its Machines card, on the same time axis as its steps. They are the whole machine's numbers, shared with whatever else ran there; the run's own peak memory and CPU are in its Resources card ([Machine metrics](https://morca.dev/docs/run#metrics)).

### Ops › Control and Ops › Pools

- **Ops › Control** (`/ops/control`, `/api/system`) is the machine running the scheduler, the UI and Postgres: CPU, memory, disk, network and each container's memory over 24 hours, 7 or 14 days, its memory pressure and whether new runs wait for it, and the local and control-placed runs going, each with its cgroup's memory against its limit.
- **Ops › Pools** (`/ops/pools`, `/api/pools`) is every other machine runs are placed on: EC2 instances, Kubernetes pods and remote workers, with each one's state and last heartbeat. With EC2 in view it also has hosts, cost and runs per hour, and **Hosts over time**: the hosts that ran during each hour, by role (EC2 heads and workers, the control host, and, since 0.23.0, self-hosted workers that ran a run or were seen that hour).

### /metrics

`/metrics` has Prometheus gauges for the watchdog (`orca_watchdog_tripped`, `_value` and `_threshold`, by rule) and for memory pressure (`orca_memory_pressure_some_avg10` and the others, and `orca_admission_memory_pressure_held`, by host).

## Where to look when a run is stuck

1. **The run's page.** A queued run's Status card says why it waits (`held` and `waiting` in its JSON): no room in the grants, memory pressure, `max_runs`, or waiting for a worker of its pool. The pipeline's page has the same under Waiting, and `orca status <pipeline>` prints it.
2. **Paused?** A watchdog trip or a deploy pauses pipelines, and a paused pipeline starts nothing. Look for the red banner, or run `orca watchdog status`.
3. **Its machine.** For a running run, the Machines card shows whether the machine is busy, out of memory or out of disk. For a pool, Ops › Pools shows whether its worker is live and when it was last heard from; a worker gone for half `ORCA_STALE_SECONDS` has its runs queued again.
4. **The host's pressure.** Ops › Control's Memory pressure card, or `memory_pressure` in `/api/workers` for a worker.
5. **The logs.** The run's driver log and its steps' logs, linked from its page.

For a run that failed rather than stalled, see [Debugging failures](https://morca.dev/docs/debug).

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.