morca

morca · API · views

orca.views.pipelines

Pipelines as the pages and their JSON read them: PipelineRow (a registration with its schedule and live runs), its steps (StepDef), its next fires, a keyed pipeline's counts (KeyCounts), its fill, and the Resources card's summary of what its runs used (UsageSummary).

Structs

PipelineRow

struct PipelineRow

Implements Copyable, Movable

Fields
name: String
registry_hash: String
binary: String
kind: String
start: String
cron: String
tz: String
fills: String
max_runs: Int
run_retries: Int
running: Int
queued: Int
last_run: Int: 0 when it has none.
last_status: String
last_partitions: String
next_fire: String
next_partition: String
paused: Bool: orca pause: the scheduler starts none of its runs.
sense_error: String: Why the sensor's last pass failed, or empty.
sense_error_at: String
placement: String: orca register --placement: control, ec2, or empty for the scheduler's runner.
cpus: String: The CPUs it asks for (orca.resources), or empty.
memory_gb: String: The memory it asks for, in GB, or empty.
waiting: String: Why its oldest queued run that waits for room on its host waits (orca.resources), or empty.
fill_lag: String: fill_lag as registered (orca.fill), or empty.

StepDef

struct StepDef

Implements Copyable, Movable

Fields
name: String
kind: String
after: String
reads: List[String]
writes: List[String]
function: String: Its function's linkage name, as describe records it.
input: String
output: String

Fire

struct Fire

Implements Copyable, Movable

Fields
at: String
local: String
partition: String

KeyCounts

struct KeyCounts

A keyed pipeline's keys and how far it is through them.

Implements Copyable, Movable

Fields
dataset: String
snapshot: Int64
total: Int
done: Int
live: Int
failed: List[String]: Keys not done or in a live run, with a failed or skipped step.
error: String: Why its keys couldn't be read, or empty.
read_at: String: When <binary> keys printed them.
note: String: Why they're older than they'd be, or empty.

UsageSummary

struct UsageSummary

What a pipeline's latest measured runs used: the median and the most of each run's peak memory, CPU seconds and peak cores; -1 for none.

Implements Copyable, Movable

Fields
runs: Int
peak_p50: Float64
peak_max: Float64
cpu_p50: Float64
cpu_max: Float64
cpus_p50: Float64
cpus_max: Float64

json

def json(self) -> String

{"runs": n, "peak_rss_bytes": {"p50": .., "max": ..}, ...}.

PipelineAsk

struct PipelineAsk

What a pipeline's page was asked for, from its URL's query: which tab (?tab=); Start runs' preview over partitions (?preview=1, &rerun=1, &allow_append=1); the month or year its coverage shows (?cal=) and which of its tabs (?cov=); the view of its steps shown without scripts (?steps=); and, in url, the page of its runs.

Implements Copyable, Movable

Fields
name: String
partitions: String
rerun: Bool
preview: Bool
cal: String
allow_append: Bool
steps: String
tab: String
cov: String
url: String

shown_tab

def shown_tab(self) -> String

overview, runs or operate: as asked, or Operate for a preview or a coverage month that names no tab.

NotRun

struct NotRun

A snapshot a pipeline's sensor saw and couldn't run, and why.

Implements Copyable, Movable

Fields
snapshot: SnapshotRef
at: String
error: String

PipelineView

struct PipelineView

A pipeline as its page and its JSON show it, read once: pipeline_json (orca.api.pipelines) and pipeline_html (orca.ui.pipeline) both draw this. The JSON is the same on every tab.

Implements Movable

Fields
pipeline: PipelineRow
steps: List[StepDef]
fires: List[Fire]: Its next five fires, if it's scheduled.
runs: RunsPage: The page of its runs url asks for.
fill: Optional[FillStatus]
following: Optional[Follow]: How far a sensor is behind what it follows (orca.follow).
usage: UsageSummary
learned: Optional[Recommendation]: The request its runs' peaks suggest (orca.recommend), from its orca config show, applied where it asks for auto.
versions: List[VersionRow]: The code it's run, newest first.
configs: List[Shown]: What orca config show <name> prints (orca.config).
keys: Optional[KeyCounts]: A keyed pipeline's counts.
coverage: Optional[Coverage]: Its partitions or keys done of those it has (orca.coverage).
not_run: List[NotRun]: The snapshots its sensor saw and couldn't run, latest first.
start: Optional[StartPreview]: What Start runs would queue, when a preview was asked for.

__init__

def __init__(out self, var pipeline: PipelineRow, var runs: RunsPage, var usage: UsageSummary)

Functions

utc_text

def utc_text(t: Int64) -> String

Seconds since the epoch as YYYY-MM-DDTHH:MM:SSZ.

comma_list

def comma_list(s: String) -> List[String]

A comma-separated list, empty for an empty string.

next_fires

def next_fires(p: PipelineRow, after: Int64, count: Int) -> List[Fire]

pipeline_rows

def pipeline_rows(mut db: State, name: String = "") -> List[PipelineRow]

Every registered pipeline, or the one named.

step_defs

def step_defs(mut db: State, name: String) -> List[StepDef]

key_counts

def key_counts(mut db: State, name: String) -> KeyCounts

pipeline_coverage

def pipeline_coverage(mut db: State, p: PipelineRow) -> Optional[Coverage]

p's coverage (orca.coverage), None if it has none or it can't be read: a list of pipelines still shows the rest.

fill_of

def fill_of(mut db: State, name: String) -> Optional[FillStatus]

name's fill (orca.fill), if it fills its range or has fill runs from when it did.

usage_summary

def usage_summary(mut db: State, name: String) -> UsageSummary

The pipeline's latest USAGE_RUNS runs that recorded what they used, summarized.

pipeline_view

def pipeline_view(mut db: State, ask: PipelineAsk) -> Optional[PipelineView]

The pipeline ask names as its page shows it, or None if there's no such pipeline.

Aliases

comptime USAGE_RUNS = 20

How many of a pipeline's latest measured runs its Resources card summarizes.