morca

morca · API

orca.follow

How far a dated sensor pipeline is behind the datasets it follows.

A sensor (orca.sensors) runs when a dataset it reads, and doesn't write, gets a snapshot, over the snapshot's partitions. What it follows has, then, is every partition of those datasets that orca knows of:

  • each snapshot a succeeded attempt of another pipeline committed (orca.commits), holding its step's partition, or the run's partitions for a gather step, as the sensor reads them;
  • each snapshot the sensor saw otherwise (orca.sensed, from <binary> sense), holding the partitions of the run it started.

Those keys, in the sensor's kind and range (from its start to today), are what it could have done; follow gives each its state as the calendar does (orca.calendar), so what's behind splits into running or queued, failed, and not queued yet. A single query reads the keys (through the commits_snapshot index), then the calendar's three.

Structs

Follow

struct Follow

Implements Movable

Fields
datasets: List[String]: What the sensor follows, by name.
upstream: List[String]: The pipelines that committed them, by name.
unit: String: day or month.
available: Int: The keys they have in the sensor's range.
done: Int
live: List[String]: Behind: in a queued, deferred or running run.
failed: List[String]: Behind: the last run over it failed.
waiting: List[String]: Behind: in no run yet.
keys: Dict[String, Bool]: Every key they have, for the calendar's behind.
pace: Pace: How long its recent runs took, to estimate queuing waiting.

__init__

def __init__(out self, unit: String)

behind

def behind(self) -> Int

Functions

follow

def follow(mut db: State, pipeline: String, kind: String, start: String) -> Optional[Follow]

How far the dated sensor pipeline (of kind, from start) is behind what it follows; None if it isn't a dated sensor.

follow_json

def follow_json(f: Follow) -> String

f as JSON: what it follows, how many keys they have and are done, and the behind ones by state, each with its count and first keys.

Aliases

comptime LISTED = 8

The most keys of each kind of behind follow_json and the page list.