morca

morca · API

orca.expiry

Snapshot expiry, the scheduler's half: what orca still needs of a dataset's history, and the binary that does the expiring.

Expiring needs iceberg, which the orca binary doesn't link, so it runs in the project's orca_maintain binary (orca.maintain). This side decides what must survive whatever the table's policy says, from Postgres:

  • the snapshots a run that can still go on has read. A step's reads are pinned: a retry or resume scans the snapshot the first attempt read, so a queued, running, deferred or failed run keeps every snapshot its attempts read. (A failed run can be resumed; a succeeded or cancelled one can't, and is let go.)
  • the snapshots such a run committed. A retry finds the commits its step made before it was killed by looking through the table's snapshots (orca.tables.find_commit); one that was expired would be committed a second time.
  • every snapshot of the last run that committed to the dataset, so an expiry never leaves fewer than one run's worth of history.

What expiry costs: orca lineage and the UI's dataset and snapshot pages read Postgres only, so they still show an expired snapshot, what made it and what it was made from; but the snapshot itself is gone from the table, so it can no longer be read (time travel, use_snapshot), and a rebuild reruns the pipelines from the current data, not from it.

Functions

retained

def retained(mut db: State, dataset: String) -> List[Int64]

The snapshots of dataset orca still needs (see the module doc).

expirable

def expirable(mut db: State) -> List[String]

Every dataset orca has committed to, by name.

maintain_binary

def maintain_binary() -> String

$ORCA_MAINTAIN, else orca_maintain beside this binary, else on PATH; empty when there is none.

expire_argv

def expire_argv(mut db: State, binary: String, dataset: String, older_than: String = "", keep: String = "", dry_run: Bool = False, report: Bool = False) -> List[String]

The orca_maintain expire command line for dataset: the policy given, or the table's own (--if-policy, so a table without one is left alone), and --retain with what orca still needs.

Aliases

comptime MAINTAIN_BINARY = String("orca_maintain")