morca · API

`orca.config`

orca.toml: one checked-in file for how each of a project's pipelines runs (#145), read by orca register and orca aws register.

[defaults] placement = "ec2" head_type = "t4g.medium"

[pipelines.gharchive]
worker_type = "c7g.xlarge"
workers = "0..4"             # or a count, 4
worker_procs = 1             # or "auto", one per vCPU
max_runs = 2                 # over the code's Pipeline max_runs
keys_per_run = 30            # over the code's: keys a run takes (#240)
cpus = "1..16"               # a request: MIN..MAX (orca.resources)
memory_gb = "2..24"          # or "auto", "auto..24" (orca.recommend)
learn = false                # don't: "auto" is then no request
env = { DISTA_WORKER_SLOTS = "4", GHARCHIVE_PARALLEL = "16" }
secrets = { GITHUB_TOKEN = "orca/github-token" }   # Secrets Manager IDs
fills = "all"                # fill the range by itself (orca.fill)
fill_budget_per_day = 5.0    # and the rest of the fill* keys

[stack]                      # for `orca aws up`; not read yet

orca config check [<file>] [--secrets] checks a file without a database, for a project's CI: what registering checks, and that orca can size each instance type (a warning for one its price table hasn't); with --secrets, that each secret can be read (its value is never printed). The pipeline page's Configuration (config_html) is orca config show's.

What stays in the pipeline's code is what it does; this file says where and how much. A key in [pipelines.<name>] beats the same key in [defaults] (env and secrets are merged name by name), and a flag of orca register beats both, recorded with the registration as an override. A pipeline the file names is registered with exactly what it and its flags say: a key taken out of the file goes back to its default. [defaults] applies only to the pipelines the file names; one it doesn't name is registered as before, each flag left out keeping what was registered.

The registration keeps the file's SHA-256 and its git commit (and whether it had changes not committed) in orca.pipelines.config, with where each value came from, for orca config show. orca config export writes the file back from what's registered.

Adding a key: a name in SCALAR_KEYS and its check in _canonical (and its types in _canonical_toml), which gives the value as text; it is then read, merged, overridden, recorded, shown and exported like the others, and only its consumer is new. The fill* keys (#155, orca.fill) were added this way, with no flag.

Structs

`Setting`

struct Setting

A value and where it came from: CODE_DEFAULT, FROM_FILE, FROM_FLAG or FROM_ENV, with a detail such as [defaults], and for the file's, the line it's on.

Implements Copyable, Movable

Fields: value: String; source: String; detail: String; line: Int: The file's line that sets it, from 1; 0 if not from the file, or registered before orca kept it.

`init`

def __init__(out self, value: String, source: String, detail: String, line: Int = Int(0))

`ProjectConfig`

struct ProjectConfig

A project's orca.toml, read and checked, or none (path empty).

Implements Movable

Fields: path: String: The file as given, for messages; empty for no file.; hash: String: The SHA-256 of its bytes, hex.; commit: String: The git commit of the checkout it's in, or empty.; dirty: Bool: Whether it differs from that commit.; source: String: Where the commit is from (ORCA_SOURCE), for a file read outside a git checkout, as in the image; empty otherwise.; repo_path: String: Its path in its checkout (git ls-files --full-name), for a link to its lines at the commit; empty outside a checkout.; doc: TomlDoc

`init`

def __init__(out self)

No file.

def __init__(out self, text: String, path: String, commit: String = "", dirty: Bool = False)

text, the file path holds, checked: every key is one this orca knows, every value of its type.

`load`

def load(path: String) -> Self

The file at path, with its checkout's commit.

`from_b64`

def from_b64(b64: String, path: String, commit: String, dirty: Bool) -> Self

The file as orca aws register sends it: base64.

`is_file`

def is_file(self) -> Bool

`has`

def has(self, pipeline: String) -> Bool

Whether the file has [pipelines.<pipeline>].

`pipelines`

def pipelines(self) -> List[String]

`where`

def where(self, pipeline: String, key: String, s: Setting) -> String

file:line: [table] of key's value s from this file, for a refusal that needs the pipeline (fills on a keyed one).

`entries`

def entries(self, key: String) -> List[Tuple[String, String]]

Each value of key in [defaults] and the [pipelines.<name>] tables, with file:line: [table] key for a message; for secrets or env, each of its names (... secrets: NAME).

`RegisterFlags`

struct RegisterFlags

What orca register's command line gave.

Implements Copyable, Movable

Fields: values: Dict[String, String]: A SCALAR_KEYS key to its flag's value, checked.; secrets: List[String]: --secret NAME=IDs, which replace the pipeline's when given.; image: String; spot: Bool

`init`

def __init__(out self)

`Resolved`

struct Resolved

What one pipeline is registered with: its settings from the file and the flags, each with where it came from.

Implements Movable

Fields: name: String; in_file: Bool: Whether the file names it; if not, flags left out keep what was registered, as before orca.toml.; values: Dict[String, Setting]; secrets: List[Tuple[String, Setting]]; env: List[Tuple[String, Setting]]; overrides: List[Tuple[String, String]]: The flags given over the file's values, key and value.

`init`

def __init__(out self, name: String, in_file: Bool)

`text`

def text(self, key: String) -> String

key's value, or empty if neither the file nor a flag set it.

`secret_specs`

def secret_specs(self) -> List[String]

`env_pairs`

def env_pairs(self) -> List[Tuple[String, String]]

`env_json`

def env_json(self) -> String

env as the JSON object Postgres keeps; empty for none.

`config_json`

def config_json(self, file: ProjectConfig) -> String

orca.pipelines.config: where each value came from, and for a pipeline the file names, the file's hash, commit and the flags that overrode it.

Functions

`flag_of`

def flag_of(key: String) -> String

The orca register flag of key: --head-type of head_type.

`parse_worker_procs`

def parse_worker_procs(v: String) -> Int

--worker-procs's value: a count of at least 1, or auto (returned as 0), one process per vCPU of the worker type.

`parse_workers`

def parse_workers(v: String) -> Tuple[Int, Int]

--workers' value, N or MIN..MAX: the fewest (-1 for a fixed count) and the most.

`parse_max_runs`

def parse_max_runs(v: String) -> Int

max_runs' count, or for auto and auto..N its ceiling (64, or N): max_runs_auto says which.

`parse_keys_per_run`

def parse_keys_per_run(v: String) -> Int

keys_per_run: how many keys a backfill, a sensor's run or a fill run takes (#240), from 1 to MAX_KEYS_PER_RUN.

`max_runs_auto`

def max_runs_auto(v: String) -> Bool

Whether max_runs' value is auto or auto..N (#266): as many runs at once as this host has room for, up to the ceiling.

`max_runs_text`

def max_runs_text(n: Int, auto: Bool) -> String

max_runs as orca.toml writes it.

`canonical_flag`

def canonical_flag(key: String, text: String) -> String

A flag's value for key, checked, as registration keeps it.

`env_name_problem`

def env_name_problem(name: String) -> String

Why name can't be a variable in env, or empty if it can.

`is_scalar_key`

def is_scalar_key(key: String) -> Bool

`is_map_key`

def is_map_key(key: String) -> Bool

`env_json`

def env_json(env: List[Tuple[String, String]]) -> String

Names and values as a JSON object; empty for none.

`parse_env`

def parse_env(text: String) -> List[Tuple[String, String]]

env_json's object back as (NAME, value) pairs.

`resolve`

def resolve(name: String, flags: RegisterFlags, file: ProjectConfig) -> Resolved

name's registration from file and flags: [defaults], then [pipelines.<name>], then the flags.

Aliases

comptime CODE_DEFAULT = "code default"

comptime FROM_FILE = "orca.toml"

comptime FROM_FLAG = "flag"

comptime FROM_ENV = "environment"

comptime SCALAR_KEYS = Array[__literal_size__=Int(16)](String("placement"), String("head_type"), String("worker_type"), String("workers"), String("worker_procs"), String("max_runs"), String("keys_per_run"), String("cpus"), String("memory_gb"), String("learn"), String("fills"), String("fill_order"), String("fill_max_runs"), String("fill_budget_per_day"), String("fill_window"), String("fill_lag"), __list_literal__=NoneType(None))

The keys of [defaults] and [pipelines.<name>] with one value. The fill* keys (orca.fill) are the file's only: no flag sets them.

comptime MAP_KEYS = Array[__literal_size__=Int(2)](String("secrets"), String("env"), __list_literal__=NoneType(None))

The keys whose value is a table of names.

comptime AUTO_MAX_RUNS = 64

max_runs = "auto"'s ceiling when it names none.

comptime MAX_KEYS_PER_RUN = 10000

The most keys keys_per_run may put in one run.