# orca.fill > A module of morca's API. Fills (#155): a pipeline filling its whole partition range by itself, within limits, set in orca.toml (`orca.config`): [pipelines.gharchive] fills = "all" # "since_registration", the default fill_order = "oldest" # or "newest" fill_max_runs = 2 # fill runs at once, beside its other runs fill_budget_per_day = 5.0 # estimated US dollars a UTC day fill_window = "22..6" # UTC hours fill runs may start in fill_lag = "2h" # how long after its end a partition fills With `fills = "all"` the scheduler queues, on each tick, runs (trigger `fill`) for the partitions of the pipeline's range that nothing has filled: from its first partition through the last one that has ended at least `fill_lag` ago, in `fill_order`, as long as every limit allows (`fill_status`). **`fill_lag`** (`0`, the default, for none) is for sources that publish a partition's data some time after it ends: GH Archive puts each hour's file up minutes after the hour, so at 00:00 UTC the day that just ended isn't all there yet. A Daily day D fills from D+1 00:00 plus the lag (on the schedule's clock), a Monthly month from the 1st of the next plus the lag. Until then the partition is outside the range, for the fill and for the coverage alike (`range_keys`), so it isn't counted as missing; the pipeline page's Fill line says when it fills (`FillStatus.lag_key`). The default, `since_registration`, is what orca did before: a schedule fills from the pipeline's registration on, and `orca backfill` the rest by hand. **What a fill takes.** A partition is left alone when it's done in a run of any code of the pipeline (every mapped step succeeded for it; with no mapped step, a run over it succeeded), when a live run has it, or when a fill run of it was ever queued, so a fill never runs a partition twice: one whose fill run failed past its retries, or was cancelled, is left to a person (`orca backfill`, or resume in the UI). Running done partitions again stays `orca backfill --rerun`. All of that is within the pipeline's current data version, its steps' `version=` (#297): raising one makes every partition fillable again, and the fill regenerates them within its limits. **Limits**, each a reason the pipeline page gives for a fill being idle: the pipeline paused (by a person or by the watchdog, `orca.watchdog`), the hour outside `fill_window`, the day's budget, the pipeline's other queued runs (a scheduled or backfilled run starts first, so fills never starve them), and `fill_max_runs`. Fill runs don't count against `max_runs`. **On a worker pool** (`placement = "worker:"`) the scheduler queues the fill runs the same way, in the same order and within the same limits, and leaves them queued, held for a worker of the pool to claim (`orca.workers`). `fill_max_runs` caps the pipeline's live fill runs across the whole pool, queued ones included, so the fill never has more than that many going whatever the number of workers; each worker's `--max-runs` and its grants cap what one machine takes. A worker pool's runs cost nothing more than its machines, which run anyway. **The cost estimate** is conservative, and fails closed. A fill run's cost is its instances' time (`orca.instances`) at `orca.fleet`'s on-demand price table, times `COST_MARGIN` for what the table leaves out (EBS, transfer, requests). The day's spend is that over the fill runs' instances since 00:00 UTC, up to now. Before starting a run, orca also counts what the fill runs going and the new one may still cost: the pipeline's hourly price (head plus its most workers) for the longest fill run of its last `_HISTORY` (an hour if it has none, at least a minute, plus the warm pool's idle time on EC2), less what each has run already. A run starts only if the spend plus that stays within `fill_budget_per_day`. When a price isn't known (an instance type not in the table, or Kubernetes, whose cluster orca can't price), no fill run starts on a budget, and the reason says so. A pipeline whose runs cost money (placed on EC2 or Kubernetes) starts no fill run without a budget. Runs on the control host or local processes cost nothing more than the machine already running. Keyed pipelines and sensors don't fill: `orca register` refuses `fills = "all"` for them (`fill_refusal`). ## Structs ### `Fill` ```mojo struct Fill ``` A pipeline's fill settings, as registered. Implements: `Copyable`, `Movable` Fields: - `all: Bool`: `fills = "all"`. - `newest_first: Bool` - `max_runs: Int` - `budget: Float64`: US dollars a UTC day; -1 for none. - `window_from: Int` - `window_to: Int` - `lag: Int`: `fill_lag` in seconds: how long after its end a partition fills. #### `of` ```mojo def of(fills: String, order: String, max_runs: Int, budget: Float64, window: String, lag: String = "") -> Self ``` From the registration's columns: empty or negative for what wasn't set. #### `in_window` ```mojo def in_window(self, hour: Int) -> Bool ``` #### `window_text` ```mojo def window_text(self) -> String ``` ### `FillPipeline` ```mojo struct FillPipeline ``` What `fill_status` needs of a registered pipeline. Implements: `Copyable`, `Movable` Fields: - `name: String` - `fill: Fill` - `kind: String` - `start: String` - `tz: String` - `paused: Bool` - `placed: String`: Where its runs would start: `control`, `ec2`, `kubernetes`, `local` or `worker:`. - `head_type: String` - `worker_type: String` - `workers: Int` - `pool_seconds: Int`: On EC2, how long an instance idles in the warm pool after a run. - `keys_per_run: Int`: How many partitions a fill run takes (#240): 1 by default. ### `FillStatus` ```mojo struct FillStatus ``` Where a pipeline's fill is, and how many fill runs may start now. Implements: `Copyable`, `Movable` Fields: - `on: Bool` - `total: Int` - `done: Int` - `live: Int`: Partitions of the range in a live run (of any trigger). - `failed: Int`: Partitions whose fill run was cancelled or failed past its retries, not done since: left to a person. - `retrying: Int`: Partitions whose fill run failed and waits for a retry. - `left: Int`: Partitions no run has taken yet. - `running: Int`: Fill runs running now. - `queued: Int`: Fill runs queued and not yet started: on a worker pool, those waiting for a worker to claim them. They count against `fill_max_runs` as the running ones do. - `rate: Float64`: Partitions an hour the fill runs did lately; 0 for none. - `eta_seconds: Int`: `left` at `rate`; -1 when there's no rate. - `spent: Float64`: Today's (UTC) estimated spend of the fill runs; -1 if unknown. - `budget: Float64` - `hourly: Float64`: A run's US dollars an hour; -1 if unknown. - `expected_seconds: Int`: How long the next fill run is expected to take. - `next_cost: Float64`: What the runs going and one more may still cost. - `idle: String`: Why no more fill runs start, or empty. - `slots: Int`: Fill runs that may start now, before the pipeline's own limits. - `next_keys: List[String]`: The partitions of each of the next fill runs, comma-separated, in `fill_order`, up to `keys_per_run` to a run; at most `slots` runs. - `lag_key: String`: The partition that has ended but waits for `fill_lag`, or empty. - `lag_until: Int64`: When `lag_key` becomes fillable, UTC seconds; 0 for none. - `lag: Int`: `fill_lag`, seconds. #### `__init__` ```mojo def __init__(out self) ``` ## Functions ### `parse_fills` ```mojo def parse_fills(v: String) -> String ``` ### `parse_fill_order` ```mojo def parse_fill_order(v: String) -> String ``` ### `parse_fill_max_runs` ```mojo def parse_fill_max_runs(v: String) -> Int ``` ### `parse_fill_budget` ```mojo def parse_fill_budget(v: String) -> Float64 ``` US dollars a day, from 0 to 100000. ### `parse_fill_window` ```mojo def parse_fill_window(v: String) -> Tuple[Int, Int] ``` `A..B`: fill runs start from hour A up to (not at) hour B, UTC; A after B wraps past midnight (`22..6`); `0..24` is all day. ### `parse_fill_lag` ```mojo def parse_fill_lag(v: String) -> Int ``` A duration in seconds: `0`, or a whole number of seconds, minutes, hours or days (`90s`, `90m`, `2h`, `1d`), up to `MAX_LAG_SECONDS`. ### `lag_text` ```mojo def lag_text(seconds: Int) -> String ``` `seconds` as `fill_lag` keeps it: `0`, `2h`, `90m` or `45s`. ### `money` ```mojo def money(x: Float64) -> String ``` `x` dollars as `$1.23`. ### `fill_refusal` ```mojo def fill_refusal(kind: String, start: String, sensor: Bool, pipeline: String) -> String ``` Why `fills = "all"` can't be registered for a pipeline, or empty. ### `last_ended` ```mojo def last_ended(kind: String, tz: String, now: Int64, lag: Int = Int(0)) -> String ``` The latest partition that had ended `lag` seconds before `now` on the schedule's wall clock (UTC without one): the end of a fill's range. The partition still going is the schedule's; one that ended less than `lag` (`fill_lag`) ago waits. ### `lag_wait` ```mojo def lag_wait(kind: String, tz: String, now: Int64, lag: Int) -> Tuple[String, Int64] ``` The partition that has ended by `now` but less than `lag` seconds ago, the oldest if several, and when it becomes fillable (its end plus `lag`, UTC seconds); `("", 0)` if none waits. ### `range_keys` ```mojo def range_keys(kind: String, start: String, tz: String, now: Int64, lag: Int = Int(0)) -> List[String] ``` Every partition from `start` through `last_ended`, oldest first. ### `run_hourly` ```mojo def run_hourly(placed: String, head_type: String, worker_type: String, workers: Int) -> Float64 ``` US dollars an hour of one of the pipeline's runs where it's `placed`: 0 on the control host, as a local process or on a worker pool's machines (which run anyway), the head and its most workers at on-demand prices on EC2 (a type left empty is the runner's, `ORCA_EC2_HEAD_TYPE` / `ORCA_EC2_WORKER_TYPE`), and -1 for a price that isn't known (a type not in `orca.fleet`'s table, or Kubernetes). ### `unknown_type` ```mojo def unknown_type(placed: String, head_type: String, worker_type: String, workers: Int) -> String ``` The instance type `run_hourly` couldn't price, or empty. ### `fill_status` ```mojo def fill_status(mut tx: Transaction, p: FillPipeline, now: Int64, run_retries: Int) -> FillStatus ``` `p`'s fill now: its progress, today's spend, whether fill runs may start and which partitions they'd take. Read in `tx`, which the scheduler holds with the pipeline's row locked. ### `lag_note` ```mojo def lag_note(s: FillStatus) -> String ``` `2026-11-08 fills from 2026-11-09 02:00 UTC (fill_lag 2h)`, or empty when no partition waits for `fill_lag`. ### `eta_text` ```mojo def eta_text(seconds: Int) -> String ``` ### `status_json` ```mojo def status_json(s: FillStatus) -> String ``` The fill as `/api/pipelines/` gives it. ## Aliases ### `ALL` ```mojo comptime ALL = "all" ``` ### `SINCE_REGISTRATION` ```mojo comptime SINCE_REGISTRATION = "since_registration" ``` The default of `fills`: a schedule fills from registration on. ### `FILL_TRIGGER` ```mojo comptime FILL_TRIGGER = "fill" ``` `orca.runs.trigger` of a fill's runs. ### `COST_MARGIN` ```mojo comptime COST_MARGIN = 1.2 ``` What the estimate adds to the price table's prices: EBS volumes, data transfer and requests aren't in it. ### `NO_HISTORY_SECONDS` ```mojo comptime NO_HISTORY_SECONDS = 3600 ``` How long a fill run is expected to take before one has ended. ### `MIN_SECONDS` ```mojo comptime MIN_SECONDS = 60 ``` EC2 bills an instance for a minute at least. ### `MAX_LAG_SECONDS` ```mojo comptime MAX_LAG_SECONDS = 2678400 ``` The longest `fill_lag`: a month and a day.