morca

morca · Docs

API reference

The orca package, from Orca 0.23.0, as published on mojoshelf (commit 32ee1ee691cc), by mojo doc from Mojo 1.2.0. Source.

orca: orchestration for Mojo data pipelines.

Declare steps and the pipeline they make, and give the binary orca's command line:

from dista import Seq
from orca import Ctx, Gather, Mapped, Month, Pipeline, Step, orca_main

def size(mut ctx: Ctx, m: Month) raises -> Int:
    return m.month * 100

def total(mut ctx: Ctx, sizes: Seq[Int]) raises -> Int:
    var t = 0
    for i in range(len(sizes)):
        t += sizes[i]
    return t

comptime Size = Step["size", Month, Int, size, Mapped]
comptime Total = Step["total", Seq[Int], Int, total, Gather, after=Size]
comptime Sizes = Pipeline["sizes", Size, Total]

def main() raises:
    orca_main[Sizes]()

See DESIGN.md for how it works and where it's going.

Modules

  • orca.actions What orca ui does to runs: each action is rows in Postgres that the scheduler, or the run's own process, acts on. 2 structs · 14 functions
  • orca.admission Admission by memory pressure, where it shows (#202): each host's newest reading in orca.pressure, for Ops › Control, /api/workers, /metrics, orca status and the home and runs pages, and the alert when a hold has nothing of orca's to wait for. 2 structs · 11 functions
  • orca.alerts What the scheduler raises alerts about (orca.notify, #184), each decided where the data is and sent through orca.notify.alert: 5 functions
  • orca.api orca's JSON API: what orca --remote, orca worker and orca.http_store.HttpStore talk to. A page's JSON is built from the same read models as its HTML (orca.views); these are the writers, and the endpoints that are JSON only.
  • orca.api.actions The actions: each a POST of a form (or of JSON, from orca --remote) that changes something, through orca.actions. 13 functions
  • orca.api.datasets Rebuilds and lineage as JSON. 2 functions
  • orca.api.ops The Ops pages' JSON endpoints: /api/fleet, /api/workers, /api/system, /api/gantt and /api/watchdog (/api/pools is the Pools page's, orca.ui.pools). /api/fleet and /api/workers are what the Fleet and Workers pages were, before Pools replaced them. 6 functions
  • orca.api.pipelines A pipeline as JSON: its registration (json_pipeline), a keyed one's counts and Start runs' preview. 6 functions
  • orca.api.runs Runs as JSON: a run (json_run), a list's page with its cursors, a code version, and a run's machines with their metrics. 7 functions
  • orca.api.worker The worker API: what orca worker (orca.worker) and the runs it starts (orca.http_store.HttpStore) call, each a POST of JSON with Authorization: Bearer <worker token>. What each call does is orca.workers; this is how a request reaches it. 3 structs · 4 functions
  • orca.auth Who may act in orca ui: users with passwords, their sessions, and a record of what each did. 11 functions
  • orca.aws orca aws: a project's orca on AWS, with the aws and docker commands its user already has (aws login, or AWS_PROFILE for a named profile). 7 functions
  • orca.aws_assets The stack template and the project Dockerfile orca aws writes out: deploy/aws/orca.yaml and deploy/project.Dockerfile, embedded by scripts/embed-aws.py. Edit those, then run it; pixi run test-aws-assets fails while this is out of date.
  • orca.aws_replace orca aws replace-control: a blue/green replacement of the stack's control instance by a new one booted from the current template, with the old one's database (DESIGN.md, "Replacing the control instance"). 1 function
  • orca.awsec2 The few EC2 calls the ec2 runner makes (orca.ec2), over objectstore's SigV4 signing and HTTP. 2 structs · 1 function
  • orca.awsjson One signed call to an AWS service's JSON API (Secrets Manager's GetSecretValue, SSM's GetParameter): SigV4 over objectstore's HTTP, with the credentials of the AWS_* environment or else the instance role. A low module: orca.control and orca.machines_ec2 both use it. 2 functions
  • orca.calendar A dated pipeline's partitions on a calendar: what state each day (or, for a monthly pipeline, month) is in, read from the database alone. 4 structs · 6 functions
  • orca.cgroup Containment (#201): a cgroup v2 memory limit per run, and retrying bigger after an out-of-memory kill. 4 structs · 20 functions
  • orca.chart Runs per day, drawn as SVG on the server for the home page: one bar a day, stacked by how the runs ended. 1 struct · 1 function
  • orca.civil Calendar arithmetic: dates to days since 1970-01-01 and back. 1 struct · 4 functions
  • orca.cli orca_main: the command line every pipeline binary gets. 3 functions
  • orca.command The orca command: the scheduler, the UI and registration, in a binary that links no pipeline. A project builds its own from two lines: 1 function
  • orca.config orca.toml: one checked-in file for how each of a project's pipelines runs (#145), read by orca register and orca aws register. 4 structs · 14 functions
  • orca.config_show orca config show and orca config export (orca.config): each pipeline's settings with where each came from, as text, HTML and JSON, and the registered settings written back as an orca.toml. 1 struct · 4 functions
  • orca.control Placement: where a pipeline's runs run (orca register --placement), and the control runner, which runs them on the scheduler's own host. 1 struct · 10 functions
  • orca.coverage A pipeline's coverage: one number for how much of it is done, shown on the home page's table, its page's Progress card and in their JSON. 1 struct · 2 functions
  • orca.crash What a crash left in a run's output: the stack dump Mojo's runtime prints on a fatal signal (a segfault, abort, a failed bounds check), cut down to a line for the error of the attempt it ended. 2 functions
  • orca.cron Cron expressions, and when they next fire in a time zone. 1 struct
  • orca.ctx Ctx: what a step gets besides its input. 1 struct
  • orca.dag A pipeline's DAG drawn as SVG, on the server. 1 struct · 2 functions
  • orca.day Day: the partition key of a daily pipeline. 1 struct
  • orca.ec2 Runs on EC2: a head instance and worker instances per run, started and ended by the scheduler (DESIGN.md, phase 5, PR 5). 2 structs · 1 function
  • orca.estimate How long runs of a pipeline would take, from how long its recent runs took. 1 struct · 3 functions
  • orca.expiry Snapshot expiry, the scheduler's half: what orca still needs of a dataset's history, and the binary that does the expiring. 4 functions
  • orca.fill Fills (#155): a pipeline filling its whole partition range by itself, within limits, set in orca.toml (orca.config): 3 structs · 18 functions
  • orca.fleet The fleet: every machine orca is paying for, as Ops' Pools shows the EC2 ones (/ops/pools?kind=ec2, orca.ui.pools) and /api/fleet has them as JSON. 6 structs · 14 functions
  • orca.follow How far a dated sensor pipeline is behind the datasets it follows. 1 struct · 2 functions
  • orca.graph A directed graph drawn as SVG, on the server: the pipelines and datasets that feed each other, and a snapshot's lineage. 2 structs · 2 functions
  • orca.hcloud Hetzner Cloud's API, as the pool autoscaler needs it (#267, orca.pool_scale): make a server, list a pool's, delete one, and the architecture of a server type. 4 structs · 3 functions
  • orca.heartbeat A run's heartbeat, kept fresh from a thread while the run's steps work. 1 struct · 1 trait · 1 function
  • orca.highlight Mojo source coloured for the UI's code-on-hover (_PEEK in orca.ui.layout), on the server, with no library: tokens splits source into runs of one kind, and highlight turns those into HTML lines with a <span> per coloured run. 1 struct · 2 functions
  • orca.hour Hour: the partition key of an hourly pipeline. 1 struct
  • orca.html Writing HTML: every value escaped, markup only where it's said to be. 1 struct · 5 functions
  • orca.http_store HttpStore: a run's state over orca ui's API, for a run on a remote worker (orca worker, orca.worker), which holds no database credentials. 4 structs · 12 functions
  • orca.instances Scheduler instances: two orca schedulers of different images overlapping on one Postgres during a deploy (docs/overlap.md). 12 functions
  • orca.json The little JSON orca writes: quoted strings. 1 function
  • orca.key Key: what a partition key is, whatever it counts. 2 traits · 1 function
  • orca.keyed Keyed partitions: keys that are names, read from a column of a dataset. 1 struct
  • orca.kube Runs on Kubernetes: a dista cluster per run, made and removed by the scheduler (K8S.md). 2 structs · 3 functions
  • orca.lineage Lineage: which snapshots a dataset's snapshot was made from, and what was made from it, from the reads and commits every attempt records. 6 structs · 6 functions
  • orca.live Which runs are live: queued, deferred, or running with their process alive. 7 functions
  • orca.locate Where a pipeline's step functions and dista tasks are defined in its source, so orca ui can link each one to its line at the commit a run ran. 3 functions
  • orca.logs Where attempts' logs, drivers' logs and runs' timelines go. 12 functions
  • orca.machines Machines for a worker pool, from whichever provider the pool names (#267): what the autoscaler (orca.pool_scale) calls, so it never knows which. 3 structs · 1 function
  • orca.machines_ec2 A worker pool's machines on EC2 (#267): orca.machines.Provider for a pool with provider = "ec2". 1 struct · 3 functions
  • orca.maintain Table maintenance for a project's datasets: snapshot expiry. 1 struct · 3 functions
  • orca.metrics A run's machine metrics on its page: the samplers' files (orca.sample) read back, their peaks and warnings, and the charts, drawn as SVG on the server like the DAG. 5 structs · 9 functions
  • orca.migrations The schema's migrations (State.migrate applies them): each version's statements, which versions raise the floor an older binary needs, and the check that an older binary may run on a newer schema. AGENTS.md says how to keep a migration additive. 4 functions
  • orca.month Month: the partition key of a monthly pipeline. 1 struct · 2 functions
  • orca.name Name: the partition key of a keyed pipeline (orca.keyed). 1 struct · 3 functions
  • orca.notify Alerts to a person: a push to the owner's phone through ntfy (#184). 2 structs · 14 functions
  • orca.partitions Partition sets: which keys a pipeline is partitioned by, from when. 3 structs · 1 trait · 1 function
  • orca.pause Pausing a pipeline: what orca pause, the UI's Pause button and the watchdog (orca.watchdog) all do. 1 function
  • orca.pipeline Pipelines: a named list of steps, checked and run in order. 1 struct
  • orca.pool An EC2 instance's side of the warm pool (orca.ec2, ORCA_EC2_POOL_SECONDS): the two commands its script runs between runs, through the pipeline's binary, over ORCA_DATABASE_URL. 3 functions
  • orca.pool_boot How a machine the pool autoscaler made becomes a worker (#267): its startup script, and the UI's answer to its /api/worker/boot. 2 functions
  • orca.pool_config Worker pools orca grows and shrinks itself (#267): a [pools.<name>] table in orca.toml, read and checked here, recorded by orca register in orca.pools, and acted on by the scheduler (orca.pool_scale). 2 structs · 3 functions
  • orca.pool_runs The scheduler's part in a worker pool (orca.workers): due retries and dead runs queued again for a worker, lost workers' runs, orphans of gone instances, and fill runs queued for the pool.
  • orca.pool_scale The pool autoscaler (#267): machines for the pools orca.toml declares (orca.pool_config), made and removed by the leading scheduler every ORCA_POOL_SCALE_SECONDS (15). 1 struct · 3 functions
  • orca.pressure Memory pressure (#202): Linux's pressure stall information (PSI) for memory, read each tick, and the admission rule built on it. 4 structs · 11 functions
  • orca.proc Starting processes: what the scheduler needs to run pipeline binaries. 2 structs · 5 functions
  • orca.queueing Queueing runs over keys: backfills, the keys a pipeline has done or holds, and what a rerun or an append refuses. 2 structs · 15 functions
  • orca.recommend Requests learned from what a pipeline's runs used (#269): shown on its Resources card and in orca config show, and applied where its cpus or memory_gb is auto (resolve_request). 3 structs · 6 functions
  • orca.registered A registered pipeline as the scheduler reads it (Registered, registered), and what its runs ask for (run_request). 1 struct · 8 functions
  • orca.registering orca register: recording a pipeline binary and how its runs run (register_pipeline). 1 function
  • orca.registry Which architectures a container image has (#267): the pool autoscaler makes a machine only of a type the worker image runs on. 2 structs · 1 function
  • orca.remote The orca command against a deployed orca ui, over HTTPS, as a user of it: log in once, then act from anywhere the UI can be reached. 3 structs · 14 functions
  • orca.resources Resource requests and grants (#171): runs sized to their share of the machine. 3 structs · 12 functions
  • orca.sample The sampler: a run's machine metrics, one JSON line every few seconds. 5 structs · 20 functions
  • orca.schedule Schedules: when a pipeline runs by itself, and which partition it fills. 4 structs · 1 trait
  • orca.scheduler The scheduler: fires schedules, senses datasets, queues backfills and rebuilds, and starts runs. 1 struct
  • orca.secrets Secrets a pipeline's runs get: orca register --secret NAME=ID. 3 functions
  • orca.sensing Fires and sensors (orca.scheduler): a schedule's fires, and a sensor's snapshots, keyed or by partition.
  • orca.sensors Runs started by datasets: sensors, and rebuilds downstream of a dataset. 3 structs · 10 functions
  • orca.source Where a pipeline binary's code came from: its git repository and commit, given at build time. 6 functions
  • orca.sqlite_store orca's runs in a SQLite file: SqliteStore, for single-node orca. 2 structs · 1 function
  • orca.starting Starting runs (orca.scheduler): due retries, dead runs and queued ones, within max_runs, memory pressure, grants and the vCPU budget; and a control-placed run's process.
  • orca.state orca's state in Postgres: pipelines, runs, steps and attempts. 4 structs · 2 functions
  • orca.step Steps: the functions a pipeline runs, and the edges between them. 3 structs · 1 trait · 3 functions
  • orca.store Where the executor records what it does. 10 structs · 2 traits · 3 functions
  • orca.system The control instance's own health: /ops/control, the gauges on /metrics, and the warning on the home page. 6 structs · 18 functions
  • orca.tables Iceberg tables from steps: pinned reads, and commits a retry recognises. 2 structs · 14 functions
  • orca.tasks The dista tasks a pipeline's steps call, and the runtime a run gets. 2 structs · 1 trait · 3 functions
  • orca.tick One pass of the scheduler (orca tick; orca scheduler makes one a second): what each registered pipeline is due, started. 1 function
  • orca.timekeys Time keys by their partitions' kind (monthly, daily, hourly), as text: for code that has a pipeline's kind as a string, not its type. 6 functions
  • orca.timeline A run's Dista timeline and its workers over time, drawn as SVG on the server for the run's page. 2 structs · 3 functions
  • orca.tokens API tokens: what orca login gets from orca ui, so the orca command on another machine can act as a user of it (orca.remote). 2 structs · 10 functions
  • orca.toml The TOML that orca.toml is written in (orca.config): a parser and a writer for the subset a configuration file needs. 3 structs · 3 functions
  • orca.tz Time zones, from the system's TZif files (RFC 8536). 2 structs
  • orca.ui orca's pages: HTML, each page in a module of its own, over the read models of orca.views; orca.web.routes says which path is which.
  • orca.ui.alerts Ops › Alerts (/ops/alerts): every alert orca.notify sent or is sending, newest first, from orca.alerts: when, its kind, its title and message as sent, links to what it is about (the run, the pipelines, the watchdog's rules, a worker, Control), whether ntfy took it, and for those that resolve (a watchdog trip, a stalled fill, the disk, memory pressure, an orca notify that was later --resolved) whether it is still open. 4 structs · 10 functions
  • orca.ui.attempt /runs/<id>/attempts/<attempt>, an attempt's page: its log, with its error, and its facts, which refresh from /facts while it runs. 4 functions
  • orca.ui.code /code?u=: a file at a commit on GitHub, its lines coloured as HTML (orca.highlight), for the code shown on hover; the UI fetches it, so a browser needn't reach GitHub. 2 functions
  • orca.ui.contained Ops > Control's Runs card: each local and control-placed run going on this host, with its cgroup's memory against its limit (orca.cgroup), which the scheduler records every 5 seconds, and its grant (orca.resources). 1 struct · 3 functions
  • orca.ui.coverage The Coverage card on a dated pipeline's Operate tab: its partitions' states (orca.calendar) as plain HTML, read-only. 7 functions
  • orca.ui.datasets /datasets, the datasets and the graph of pipelines and datasets; /datasets/<name>, a dataset's writers, readers, snapshots and what a rebuild of it runs; /datasets/<name>/snapshots/<id>, a snapshot's lineage (orca.lineage); /rebuilds/<id>, a rebuild's runs by level. 6 functions
  • orca.ui.home /, the home page: the pipelines, with their coverage (orca.coverage), schedules, next and last runs, the recent runs, and the Timeline tab (orca.ui.viz); /home/pipelines is its table of pipelines, as the fragment it refreshes from. 2 functions
  • orca.ui.layout Every page's frame (page_html): its head, the nav, the slots the server fills in with who's logged in (orca.web.server), and the scripts every page has (_POLL for live fragments, _TABS, _TIPS, _PEEK for code on hover); and what every page draws with: cards, facts, help, action buttons, and the 404 and 400 pages. 12 functions
  • orca.ui.login The login page (/login, orca.web.auth). 2 functions
  • orca.ui.machines A run's machines on its page: each instance with its role, its metrics' peaks and warnings (orca.metrics), and the steps that ran on it. 2 functions
  • orca.ui.ops The Ops pages: Control (/ops/control, the machine running the scheduler, the UI and Postgres; orca.system), the watchdog (/watchdog; orca.watchdog) and Prometheus' /metrics, with the fragments each refreshes from; Alerts (/ops/alerts, where /ops goes) is orca.ui.alerts and Pools (/ops/pools) is orca.ui.pools. Fleet and Workers' old paths redirect to Pools (moved, to_pools). 8 functions
  • orca.ui.parts Pieces several pages draw: durations, times and partitions in a table's cells, a runs table with each row's actions, links to code, and a run's grant and usage as facts. 18 functions
  • orca.ui.pipeline /pipelines/<name>, a pipeline's page: Overview (its cards: Progress, Steps, Code, Resources, Configuration), Runs and Operate (Start runs, cancel, pause), each a tab of its own; /progress, the Progress card as the fragment it refreshes from; and /keys/<key>, one key of a keyed pipeline in every run it was in. 4 functions
  • orca.ui.pools Ops' Pools (/ops/pools): a row per machine of every pool runs are placed on (orca.views.pools), filtered by pool and kind with plain links, refreshed every 10 s from /ops/pools/live; a machine's own page (/ops/pools/<kind>/<id>); and /api/pools. With EC2 in view the page keeps Fleet's summary cards and charts (orca.fleet). 1 struct · 8 functions
  • orca.ui.run /runs/<id>, a run's page: its cards (Status, Partitions & trigger, Code, Resources, Settings), its steps as a table, a timeline or a graph, its machines, its snapshots and who acted on it; and its fragments and files: /live, /steps, /metrics, timeline.json, driver.log and each machine's log. 7 functions
  • orca.ui.runs /runs, the runs list: newest first, filtered by pipeline, status (any of several, each with its count), trigger, when it was created, a run and its partitions, a page at a time (orca.views.runs.RunsPage), and its runs picked to cancel, retry or backfill again in one go (orca.api.actions.bulk); /runs/table is a list's page as the fragment it refreshes from, for /runs, a pipeline's Runs tab and the home page. 10 functions
  • orca.ui.start Start runs, on a pipeline's Operate tab: the form, its date picker (orca.ui.coverage), and the preview of what it would queue (orca.views.start), updated as you type. 1 function
  • orca.ui.step_cards A card per step, on hover, keyboard focus or a tap, in the UI's steps views: a pipeline's Steps card (Graph and Table) and a run's (Table, Timeline and Graph). 3 structs · 9 functions
  • orca.ui.style The stylesheet orca ui serves at /static/orca.css: magmalake.dev's colours and type, light and dark, and system fonts (nothing is fetched).
  • orca.ui.terms What the UI's words mean, in one place: the help on every table's column headers, and on the values that come from a fixed set (statuses, step kinds, triggers, placements, where a setting came from), so the same word has the same explanation on every page. 3 functions
  • orca.ui.viz Two views orca ui draws in the browser: a run's Dista timeline, and the pipelines over time on the home page. Both are plain scripts in the page (nothing is fetched from elsewhere) over JSON the UI serves, and both load that JSON only when they're looked at, so the pages stay as fast as they were. 5 functions
  • orca.upgrade A release's upgrade steps (#323): what has to be done before or after a deploy, which a project's image ships in upgrade.toml and the Hetzner deploy shows or runs (deploy/hetzner/deploy.sh). 1 struct · 6 functions
  • orca.usage What a run's processes use: each attempt's peak memory and CPU. 3 structs · 9 functions
  • orca.views What the pages and their JSON show, read from the database once into typed rows: the query layer orca.ui and orca.api share.
  • orca.views.code Where a pipeline's code is: each step's line at the commit a run, or the registration, was built from (CodeLinks), for the link beside every step a page shows. 3 structs · 4 functions
  • orca.views.commands The orca command lines the pages show beside their buttons, so what the UI does can be done from a shell, quoted for sh. 7 functions
  • orca.views.datasets Datasets as their pages read them: the steps that write and read one, and what a rebuild of it would queue (RebuildPreview). 2 structs · 3 functions
  • orca.views.pipelines Pipelines as the pages and their JSON read them: PipelineRow (a registration with its schedule and live runs), its steps (StepDef), its next fires, a keyed pipeline's counts (KeyCounts), its fill, and the Resources card's summary of what its runs used (UsageSummary). 8 structs · 10 functions
  • orca.views.pools Ops' Pools (/ops/pools): every machine runs are placed on, one row each, by the pool a placement names. 3 structs · 8 functions
  • orca.views.runs Runs as the pages and their JSON read them: RunRow with its grant and what it used, its steps (StepRow), the tables they read and wrote (Touch), its machines (InstanceRow) and code (VersionRow), a step's attempts (AttemptRow), and a runs list's page (RunsQuery, RunsPage: keyset by run id, so pages stay put as runs arrive). 12 structs · 24 functions
  • orca.views.start What Start runs would queue (StartPreview), as a sentence and as the command that queues it, for a pipeline's Operate tab and its JSON. 1 struct · 4 functions
  • orca.views.steps What the step cards show (orca.ui.step_cards): each step's card, with what its attempts in the pipeline's latest runs did, and on a run's page each attempt's. 2 functions
  • orca.warehouse A dataset's current snapshot, read from its table's metadata file without iceberg, for <binary> sense. 1 struct · 2 functions
  • orca.watchdog The watchdog: pauses pipelines when something leaves a sane range. 8 structs · 41 functions
  • orca.web orca's web server: one router over the pages (orca.ui) and the JSON API (orca.api), who each request is from, and the listeners.
  • orca.web.auth Who a request is from, and whether it may act: the session cookie of a browser, the API token of a script (orca --remote), and the checks that keep another site's page from posting to the UI. Also the endpoints that log in and out. The users, sessions and audit rows themselves are orca.auth; API tokens are orca.tokens; a worker's token is checked by orca.api.worker instead, and gets nothing here. 17 functions
  • orca.web.drain Stopping orca ui without dropping a request. docker stop sends SIGTERM and waits 10 s before it kills; Ctrl-C sends SIGINT. Either one starts a drain: 2 structs · 1 function
  • orca.web.page What a handler answers, Page: its status, its JSON and its HTML, where to send a browser, and respond, which sends it: the HTML, or the JSON to a script; a fragment or a file as itself, whatever was asked for. 1 struct · 1 function
  • orca.web.router The router: a table of routes, each a method, a path pattern, what kind of request it is and the handler that answers it. orca.web.routes is the table; orca.web.server looks a request up here and wraps the handler in what its kind needs. 3 structs · 5 functions
  • orca.web.routes Every path orca ui answers, and what answers it: the route table orca.web.server serves (orca.web.router for how a path matches). 1 function
  • orca.web.server orca ui's server: one handler that looks each request up in the route table (orca.web.routes), checks who it's from (orca.web.auth) and answers it as its route's kind says. 1 struct · 3 functions
  • orca.worker orca worker: runs a pool's runs on this machine, talking only to orca ui's API. 1 struct · 1 function
  • orca.workers Remote workers, the server's side: worker tokens, claims, and the store calls a worker's runs make over orca ui (orca.http_store). 3 structs · 15 functions