morca

morca · API

orca.pool_scale

The pool autoscaler (#267): machines for the pools orca.toml declares (orca.pool_config), made and removed by the leading scheduler every ORCA_POOL_SCALE_SECONDS (15).

Demand. A pool needs a machine while runs of its pipelines (placement worker:<pool>, not paused) have been queued ORCA_POOL_GROW_AFTER seconds (20) that a worker could start: no more of a pipeline's than max_runs less its running ones, and its fill runs. Workers already in the pool take what they have room for first; what's left waiting is the demand.

Growing. While there's demand, no machine of the pool is still booting (made, its worker not yet seen) and it has fewer than max_machines, the autoscaler makes one, at most once every ORCA_POOL_TRY_SECONDS (60):

  • it tries the type it's on (the first, to start with) in each location in turn; the first that the provider has is made;
  • a type the worker image has no variant for (orca.registry: the machine's architecture, as the provider says it, isn't among the image's) is skipped, saying so;
  • when no location has the type, it keeps trying it until the type's wait has passed since it started, then moves to the next; the last has no wait and is tried for as long as there's demand;
  • once one is made, or the demand goes, it starts again from the first.

The machine is named orca-<pool>-<n>, labelled with the pool and this stack's id (orca.stack_identity, so a project several stacks share keeps them apart), and booted as a worker (orca.pool_boot) with a token of its own. orca.pools.status says what the last attempt came to.

Shrinking. A machine is removed (its token revoked, then the machine deleted, then its row ended with why) when:

  • it has run nothing for its pool's idle_seconds and the pool has no demand;
  • its pool is no longer in orca.toml, once it's idle;
  • its worker never showed up within ORCA_POOL_BOOT_SECONDS (900), or went silent for ORCA_POOL_LOST_SECONDS (300): its runs are queued again as any lost worker's are (orca.pool_runs).

Strays. A machine of the provider's with the pool's and this stack's labels that orca.pool_machines doesn't have live (made, but the row wasn't written) is deleted.

Structs

PoolScaler

struct PoolScaler

The autoscaler's state between passes: when it last ran, this stack's id, and the worker image's architectures (read once per image).

Implements Movable

Fields
last: Float64
owner: String
image: String
arches: List[String]
arches_why: String
arches_at: Float64: When they were read (perf_counter_ns seconds): a tag can be pushed again, so they're read again after ORCA_POOL_IMAGE_SECONDS (600).

__init__

def __init__(out self)

due

def due(mut self, now: Float64) -> Bool

scale

def scale(mut self, mut db: State)

One pass over every pool.

Functions

scale_seconds

def scale_seconds() -> Float64

stack_id

def stack_id(mut db: State) -> String

This stack's id, made once (orca.stack_identity): the label that marks its machines as its own.

demand

def demand(mut db: State, pool: String, after: Float64) -> Int

The runs of pool's pipelines queued at least after seconds that a worker could start: per pipeline at most max_runs less its running ones, and its fill runs.