morca

Operations

Upgrades and rollbacks

An upgrade is a new image. morca keeps the database working for the image before it, so going back is another deploy rather than a restore. This page covers releases, the deploy, the schema rules that make rollbacks cheap, backups, and the workers.

Releases and images

Each morca release has a version, the orca-mojo version a project pins, and an entry in CHANGELOG.md. With it come:

  • ghcr.io/magmalake/orca-runtime:<version> (from 0.24.0), for linux/arm64 and linux/amd64, with orca, its tools and the /opt/orca layout;
  • orca-hetzner.tar.gz, the deploy scripts, which print the version they came with (The scripts).

A project's image builds FROM the runtime of the morca version it pins and adds its own pipelines. Images are tagged by the project's short commit sha, not by version, so moving a project to a new morca is a commit that bumps the pin, and its image is the thing you deploy (Images).

Deploying a new image

./deploy.sh a1b2c3d                    # the image at that tag; starts again what changed
./deploy.sh --rollback                 # back to the tags deployed before

Deploying has what a deploy replaces and how the UI and apps change without downtime. Three parts matter for an upgrade:

  • A backup first. Each deploy that replaces anything backs the database up just before it applies, and prints the dump's key (since 0.16.0).
  • Upgrade steps (since 0.23.0). A release that needs something done around its deploy, such as a backfill after it or a new secret before it, lists it in its image's /opt/orca/upgrade.toml. When the morca tag changes, deploy.sh reads the new image's before steps first and stops until each is done, then runs the after steps once the pipelines are registered and before they are unpaused. orca upgrade --from <old> --to <new> prints the steps between two versions, skipped versions' included.
  • The check after it. Every deploy, a rollback too, ends with orca-stack check: Postgres answers, each container runs the image it should, there is one scheduler, every pipeline's binary is in the image, and, on a dev stack, CANARY runs succeed. It undoes nothing; a failed check means --rollback is the way back (The check after each deploy).

Additive migrations

The new image migrates the schema when its scheduler or UI starts. Postgres is never rolled back with the image, so the older image has to run on the newer schema. It can, as long as every migration it doesn't know is additive. morca's own rule for its migrations (AGENTS.md, "Migrations"):

  • add columns only nullable or with a default;
  • add tables and indexes freely; a foreign key onto a table morca deletes from needs ON DELETE CASCADE or SET NULL;
  • don't rename, drop or retype anything, tighten a constraint, change a key an ON CONFLICT names, write a status older code acts on, or give an old column a new meaning.

Queries name their columns and read rows by name, so a new column never shifts what older code reads.

An older binary on a newer schema

When a binary starts on a database ahead of it, it checks the schema's floor: the fewest migrations a binary must know to run on it, which is the newest migration listed in _floor_raisers in src/orca/migrations.mojo. If the binary knows at least that many, it logs that it is running on a newer schema and goes on: 

orca: the database is at schema version 41, ahead of this binary's 40; its migrations since are additive, so running on it

If not, it refuses to start, naming the version it needs. The floor raisers so far are migration 1, which creates the schema, and 24, the EC2 warm pool's change to orca.instances' key. This holds from morca 0.16.0, the first that records floors: an image with an older morca stops at any newer schema, additive or not.

Floor-raising migrations

A migration that has to break the rule adds its version to _floor_raisers, saying why. From then on, a rollback to an image from before it needs the database restored from the backup taken before the upgrade, and a deploy across it can't overlap schedulers.

Rolling back

Each deploy keeps the tags it replaced. ./deploy.sh --rollback applies the previous morca and app tags and, if it replaced the scheduler, registers the pipelines again from the older image, so runs start with the code that is deployed. A second rollback goes forward again. Rolling back only an app leaves the scheduler alone (Rollbacks).

Across a floor-raising migration, restore the backup, then roll back: 

ssh root@<ip> orca-stack list-backups      # the dumps, by time
ssh root@<ip> orca-stack restore --force backups/pg/20261108T101500Z.dump
ssh root@<ip> orca-stack rollback --force  # the old images on it, registered again

The restore unpauses the pipelines the deploy paused and leaves those paused by hand or by the watchdog paused. What happened in morca's database since the dump is lost: runs, registrations, users and tokens added. The warehouse isn't in the dump, so snapshots committed since stay in the tables (Rolling back).

A run with steps done by the newer code fails after a rollback, saying how to resume it with --allow-new-code. And register without "auto" requests before rolling back to a morca before 0.21.0, which refuses them.

Backups

The Hetzner server runs orca-stack backup every night at 02:30 UTC: a pg_dump of the orca schema, checked to read back, uploaded to s3://<bucket>/backups/pg/<time>.dump, keeping the newest 14. Beside each dump is a file saying which pipelines were paused by hand and which by a deploy. A failed backup sends an alert. 

./backup.sh                               # now, e.g. before a risky step
./backup.sh list
./up.sh --restore latest                  # onto a new server and an empty Volume

Only morca's own state (runs, pipelines, users) is in the dump. The warehouse is in the bucket and needs no restore. Test a restore before you need one (Backups and restore).

Overlapping schedulers

Restarting the scheduler kills the local runs it holds, so by default a deploy that replaces it pauses the pipelines and waits for those runs. With orca-stack configure OVERLAP=1 (since 0.18.0, off by default) it doesn't. The new scheduler starts beside the old one on the same Postgres:

  1. orca scheduler can-overlap, from the new image, says yes only if a live scheduler leads and every live one knows at least the new binary's floor. On no, the deploy pauses and waits as before, saying why.
  2. The new scheduler migrates (additively) and joins. The newest live scheduler leads, by a Postgres advisory lock, so the old one gives up the lead and drains: it starts nothing more and finishes its runs under its own image.
  3. The deploy waits for the new one to lead (orca scheduler wait-lead), then registers the pipelines, so new runs start with the new code.
  4. The old scheduler exits once its runs end, or after ORCA_DRAIN_SECONDS (3600) kills what is left; those runs are queued again.

A draining scheduler never takes the lead back. If the new one dies first, nothing starts until a newer one runs: the container's restart, or a rollback (overlap.md).

Workers on the same tag

A remote worker runs the code of its own image, so keep it on the server's tag. Deploy the server first, then restart the worker on the same tag. A server deploy doesn't wait for a worker's runs, and they go on to the new UI (Remote workers).

If the pipelines are registered with --image, a worker of another image is given none of their runs, which stay queued saying why, so nothing runs on a mismatched worker. Without --image, a worker still on the old tag runs the old code until it is restarted. Machines of an autoscaled pool run the image the UI gives them when they boot (ORCA_WORKER_IMAGE, the server's own), so machines made after a deploy get the new one; a machine already up keeps the image it booted with until it is removed.