morca

Docs

Deploying

On one Hetzner server, a deploy shouldn't cost a run, and a bad one should be easy to undo. The scripts that do it come with each morca release. This page is the outline; docs/hetzner.md in the repository is the full runbook. On AWS, orca aws makes and updates the stack.

The scripts

Since 0.19.0 the deploy tooling is a release asset, so deploying needs no checkout of morca's repository. It unpacks to orca-hetzner-<version>/, and the scripts print that version: 

curl -fsSL https://github.com/morca-dev/morca/releases/latest/download/orca-hetzner.tar.gz | tar -xz
cd orca-hetzner-<version>
cp hetzner.env.example hetzner.env     # fill in every line
./up.sh --dry-run                      # prints every hcloud and ssh command; runs none
./up.sh

up.sh makes the firewall, the Volume and the server, puts the settings and secrets on it and starts the containers: Postgres, the scheduler, the UI, the sampler, the apps and the Cloudflare tunnel (First setup).

Deploys

./deploy.sh a1b2c3d                    # the image at that tag; starts again what changed
./deploy.sh a1b2c3d console=latest     # one app on another tag
./deploy.sh --all a1b2c3d              # every container again, changed or not

Only what changed starts again (since 0.18.0). orca-stack apply compares each container's image, env file, flags and mounts with what it runs, and leaves it alone when nothing differs; orca-stack plan says what an apply would do. A census-only deploy replaces the console and nothing else, and Postgres is never restarted.

The UI and the apps without downtime (since 0.18.0). A new UI or app starts beside the old one, takes its name on the network once it's healthy (the UI's /healthz), and the old one finishes its requests and stops. One that never gets healthy is removed and the old one keeps serving. Remote workers talk only to the UI, so their runs go on through a deploy.

The scheduler. Restarting it kills the runs it holds, so when the plan replaces it, deploy.sh pauses the pipelines, waits for the scheduler's running runs (not those on remote workers), backs the database up, applies, registers the pipelines from the new image and unpauses what it paused. When the plan doesn't replace the scheduler, nothing is paused.

Apps

From 0.26.0, an app's setup lives with its code. The stack names the apps it runs, and the app's own image says what each one gets, in its [apps.<name>] table. The deploy script no longer hard-codes any of it: 

APPS=console                      # which apps this stack runs
SECRETS_FILE=secrets.sops.yaml    # holds secrets/orca/posthog-key, if the app asks for it
  • Read from the image. morca-stack plan and apply run the app image's morca config apps, with the stack's overlay laid on it, and write the app's env file (/etc/orca/app-<name>.env, mode 600) from what it declares. That file holds its env, the lake with lake = true, each secret and PORT; port, check and memory_gb set how the container runs.
  • Secrets. A secret the app asks for, orca/<id>, is a key secrets/orca/<id> in SECRETS_FILE, sent to the server like the stack's other secrets. If the server doesn't have one, the deploy stops at the plan, before anything is paused or replaced. Values are never printed, and changing one replaces the app, without downtime.
  • Before 0.26.0. APPS=console:8080 still works: the port is used for an app whose table gives none, and a port that differs from the table's is refused. An app the image doesn't declare, or an image from before 0.26.0, gets the fixed env of before, the lake and the UI's PostHog key as POSTHOG_KEY, and the plan says so.
  • Per stack. What one stack runs differently, such as an app's env on dev, goes in that stack's overlay: a morca.toml-shaped file named by CONFIG_OVERLAY_FILE in hetzner.env, checked in the new image before a deploy changes anything (A stack's overlay).

The full runbook is Apps in docs/hetzner.md (#331, #332).

Overlapping schedulers

Since 0.18.0, with orca-stack configure OVERLAP=1 (off by default), a deploy that replaces the scheduler doesn't pause. The new scheduler starts beside the old one and takes the lead, and the old one starts nothing more, finishes its runs under its own image and exits (#232). A migration that raises the schema's floor refuses the overlap, and the deploy then pauses and waits as above, saying why.

Rollbacks

./deploy.sh --rollback                 # back to the tags deployed before
ssh root@<ip> orca-stack apply --app console=<sha>   # only an app

Each deploy keeps the tags it replaced and, when it replaces anything, a database backup first (since 0.16.0). --rollback applies the previous tags and, if the scheduler was replaced, registers the pipelines again, so runs start with the code that's deployed. Postgres isn't rolled back: an older morca runs on a newer schema whose migrations since are additive. Only a migration that raises the floor needs the backup restored (Rolling back).

Settings and secrets

Settings from hetzner.env (since 0.18.0). Every deploy, rollbacks too, brings the server's settings and secret files in line with hetzner.env and prints the names of what changed, never a value. So hetzner.env is where a setting changes: a hand edit to /etc/orca/settings.env is undone by the next deploy, and an extra line belongs in EXTRA_ENV_FILE.

Secrets encrypted with SOPS (since 0.19.0). A secret file that hetzner.env names as *.sops or *.sops.<ext> is decrypted with sops -d on its way to the server and never written to a file on the laptop, so the laptop keeps one age key instead of plain tokens (Secrets on your laptop). One SECRETS_FILE can hold every secret, decrypted once, so a Secure Enclave key asks for Touch ID once per deploy (#265).

Rescaling

./rescale.sh up                        # CAX21 -> CAX41, keeping the disk
./rescale.sh down                      # CAX41 -> CAX21

For an import, rescale.sh changes the server's type. It pauses every pipeline and waits for the running runs, those on remote workers too, since the UI they talk to goes down with the server; backs up, changes the type, waits for the scheduler and unpauses what it paused. The server is down for a few minutes (Rescaling for an import).

Next

  • morca #247: orca deploy in the CLI, with the scripts built into the binary and the settings in the app's repository.
  • morca #250: a dev stack and a release flow that promotes the image tested there to production.
  • morca #263: deploying on GCP's ARM machines beside Hetzner, and the scripts working against any host with SSH.