# orca.heartbeat

> A module of morca's API.

A run's heartbeat, kept fresh from a thread while the run's steps work.

Steps run on the main thread and can take minutes, so the heartbeat can't
wait for the next attempt to be recorded. A `RunStore` keeps one as a
`Heartbeat[B]`: one thread calls `B.beat` every `interval` seconds, on a
connection of its own, until the heartbeat is stopped or dropped, which
joins it. A failed beat is retried on the next one, after `B.reset`: a run
whose heartbeat goes stale can be resumed elsewhere, and this process then
finds out at its next attempt (see `orca.state`).

**Stopping a step.** A step runs on the main thread and can't be
interrupted there, so the heartbeat thread is also the run's watchdog: on
each beat the store's `Beater` (`orca.state.PgBeat`) also looks for an
attempt of a step declared `timeout_seconds=N` that started more than N
seconds ago, and for a cancel (`runs.cancel_requested`) whose `cancel_at`
has passed, set to `ORCA_CANCEL_KILL_SECONDS` after the cancel by whoever
asked. For the first, the attempt and its step are `failed` ("timed out
after N s") and the run `failed`, with that as its error, so the scheduler
retries it as it retries any failed run; for the second, the running
attempt is `failed` ("cancelled") and the run `cancelled`. Each is one
transaction, with an `orca.audit` row (user `orca`, action `stop`) saying
why, and then the process exits at once, with code 1, taking a local run's
dista head and workers with it. On Kubernetes the Job's pod ends, and on
EC2 the head's script shuts its instance down; either way the scheduler's
sweep removes the rest of the run's cluster, as for any run that isn't
running. A commit the step made before it was stopped is recognised by the
attempt that runs it again (DESIGN.md, "Snapshots"). `SqliteStore`'s beater
stops a step past its timeout the same way, against its file.

## Structs

### `Heartbeat`

```mojo
struct Heartbeat[B: Beater]
```

A thread beating `B` until this is dropped.

Implements: `Movable`

#### `__init__`

```mojo
def __init__(out self, var beater: B, interval: Float64)
```

#### `__deinit__`

```mojo
def __deinit__(deinit self)
```

## Traits

### `Beater`

```mojo
undefined
```

Refreshes one run's heartbeat in a store.

Implements: `Movable`

#### `beat`

```mojo
def beat(mut self)
```

Refreshes the heartbeat once.

#### `reset`

```mojo
def reset(mut self)
```

Drops the connection a failed beat used, so the next one opens another.

## Functions

### `heartbeat_interval`

```mojo
def heartbeat_interval(stale_seconds: Float64) -> Float64
```

Five beats per stale limit, so a few missed ones don't lose the run.