orca.cli

taxi describe                              the DAG as JSON
taxi run --partitions 2024-01..2024-03     starts a run and runs it
taxi run ... --allow-new-code              ... with code the scheduler wasn't given
taxi resume <run> [--allow-new-code]       continues a run at what's unfinished
taxi status [<run>]                        the latest runs, or one run's steps
taxi status --before <run>                 the 20 runs older than that one
taxi status --status failed,queued         the latest runs with one of those statuses
taxi sense                                 the current snapshot of each dataset it reads
labels keys                                a keyed pipeline's keys snapshot, then its keys

For a keyed pipeline (orca.keyed), --partitions takes names: a list, @all or @file:<path>. A name its keys dataset doesn't have now is refused, as a malformed key is, with exit code 2.

With ORCA_DATABASE_URL set, runs are recorded in Postgres and can be resumed, or, for sqlite:<path> (sqlite:///abs/path for an absolute one), in that SQLite file (orca.sqlite_store); without it, run records its run in memory (MemoryStore, run 0) on the same path, and resume and status aren't available. With ORCA_STATE_URL (and ORCA_STATE_TOKEN) instead, as orca worker starts a run, resume, status <run> and keep-log record over orca ui's API (orca.http_store), and run is refused. run records the pipeline the first time; once orca register has given it a binary, run leaves the registration as it is, since what the scheduler queues and counts as done is that code's. A binary whose code isn't the registered one's is refused (exit code 2) unless run is given --allow-new-code; its run then counts as that code's, not the pipeline's. ORCA_STALE_SECONDS (default 30) is how old a run's heartbeat must be before another process may resume it.

A pipeline with dista tasks gets a runtime per run, started before anything else so its forked processes inherit no connection or thread; run with DISTA_ROLE set, the binary is a dista head or worker instead (see orca.tasks). With ORCA_LOGS set, a directory or a location in object storage (orca.logs), attempts log there, the run's dista timeline is written beside them, and status <run> shows each step's latest log.

sense prints a line per dataset the pipeline reads and doesn't write, <dataset> <snapshot> <orca|other> <partitions|->: its table's current snapshot in ORCA_WAREHOUSE (-1 for none), whether an orca step made it, and the partition keys its summary says it holds. The scheduler polls it for a sensor pipeline (orca.sensors).

run and resume print one line per step and partition, then the run's status. A run whose step raised defer (and none failed) ends deferred, and the scheduler starts it again when the step asked. Exit codes: 0 when the run succeeded, 1 when a step didn't, the run deferred or was cancelled, 2 for a usage error, 3 for any other error (the database, a run that can't be resumed).