morca

morca · API

orca.sqlite_store

orca's runs in a SQLite file: SqliteStore, for single-node orca.

A pipeline binary takes ORCA_DATABASE_URL=sqlite:<path> (or sqlite://<path>, so sqlite:///var/orca/runs.db is an absolute path): run, resume and status then keep pipelines, runs, steps, attempts, reads and commits in that file, through sqlite-mojo, with nothing to operate. A run killed anywhere is resumed from the file as from Postgres, by the same RunStore code path. The scheduler, the UI and orca's other commands stay on Postgres (orca.state.State refuses a SQLite URL).

The tables are Postgres's, under the same names without the orca. schema, holding what a pipeline binary writes: steps' inputs and outputs are hex text, times are Unix seconds (julianday based, to the millisecond).

Concurrency. The file is in WAL mode, so readers (status, sqlite3) don't block the writer and the writer doesn't block them. There is one writer at a time: each change is one BEGIN IMMEDIATE transaction, which takes the file's write lock up front, and a writer that finds it taken waits up to a second (busy_timeout), then fails; the owned run's writes are retried, reopening the file, for ORCA_DB_RETRY_SECONDS, as PgStore reconnects. That serialises several runs' processes on one machine, which is fine for steps that take seconds and rows that take microseconds; it isn't meant for hundreds of runs at once. Claims are a single guarded UPDATE, so two resumes racing for a run can't both win, and every attempt checks ownership in its own transaction, as on Postgres. The file must be on a local disk: SQLite's locks don't hold over NFS and most network file systems, so two machines must not share one.

synchronous = FULL: a commit is on disk before it returns, so a power loss, not only a killed process, keeps every recorded attempt.

Structs

SqliteBeat

struct SqliteBeat

Refreshes a run's heartbeat in the file, on a connection of its own.

Implements Beater, Movable

Fields
path: String
run: Int
owner: String
timeouts: String: step=seconds for each step with a timeout, comma-separated.

__init__

def __init__(out self, path: String, run: Int, owner: String, timeouts: String = "")

beat

def beat(mut self)

reset

def reset(mut self)

SqliteStore

struct SqliteStore

orca's runs in a SQLite file, as their owner.

Implements Movable, RunStore, Store

Fields
path: String
db: Database
run: Int
owner: String
registry_hash: String

__init__

def __init__(out self, path: String)

Opens the file at path, creating it if there's none; own a run before recording attempts.

migrate

def migrate(mut self)

register

def register(mut self, pipeline: String, registry_hash: String, describe: String)

scheduled_hash

def scheduled_hash(mut self, pipeline: String) -> String

version

def version(mut self, pipeline: String, registry_hash: String, describe: String, image: String) -> Int

create_run

def create_run(mut self, pipeline: String, registry_hash: String, partitions: String, trigger: String, owner: String) -> Int

claim

def claim(mut self, run: Int, owner: String, stale_seconds: Float64)

ran_version

def ran_version(mut self, run: Int, version: Int)

interrupt

def interrupt(mut self, run: Int, why: String = "") -> Int

prior

def prior(mut self, run: Int) -> RunResult

run_info

def run_info(mut self, run: Int) -> RunInfo

runs

def runs(mut self, limit: Int, before: Int = Int(0), status: String = "") -> List[RunInfo]

steps

def steps(mut self, run: Int) -> List[StepInfo]

own

def own(mut self, run: Int, owner: String, registry_hash: String)

start_heartbeat

def start_heartbeat(mut self, interval: Float64, timeouts: String = "")

stop_heartbeat

def stop_heartbeat(mut self)

finish_run

def finish_run(mut self, status: String)

defer_run

def defer_run(mut self, until: Int64)

cancel_requested

def cancel_requested(mut self) -> Bool

start_attempt

def start_attempt(mut self, step: String, partition: String, ord: Int, input: Span[UInt8]) -> Int

finish_attempt

def finish_attempt(mut self, step: String, partition: String, attempt: Int, status: String, output: Span[UInt8], error: String)

skip

def skip(mut self, step: String, partition: String, ord: Int, reason: String)

run_token

def run_token(mut self) -> String

pinned_reads

def pinned_reads(mut self, step: String, partition: String) -> Dict[String, Int64]

record

def record(mut self, commit: Bool, t: TableSnapshot)

fence

def fence(mut self)

set_log

def set_log(mut self, step: String, partition: String, attempt: Int, path: String)

Functions

sqlite_path

def sqlite_path(url: String) -> String

The file of sqlite:<path> or sqlite://<path>, or empty if url isn't a SQLite URL.