orca.aws_replace

orca aws replace-control [--stack S] [--plan] [--wait] [--keep-old 24h] [--smoke PIPELINE=KEY] [--env KEY=VALUE ...] orca aws replace-control --resume # after a failure, or to retire orca aws replace-control --rollback # back to the old instance

The control instance is in one of two slots, blue and green (deploy/aws/orca.yaml); the Elastic IP is on the active one. The steps, each logged to build/orca-aws/replace-<stack>.log and recorded in build/orca-aws/replace-<stack>.state once done, so --resume goes on from the first one that isn't:

1. quiet: every pipeline paused and no run running, else it refuses, printing which; --wait waits for the runs (it pauses nothing).
2. freeze: the old instance's scheduler and UI stop (and won't restart with it), so nothing writes to its database; the UI is down from here.
3. backup: pg_dump -Fc of orca's schema to s3://<bucket>/backups/<stack>/<UTC time>.dump, verified by restoring it into a scratch database and comparing pipelines, runs, the last run and the schema version; beside it the image tag, Caddy's data (its certificates), the apps (orca-app), the non-secret settings (/etc/orca.env without URLs with passwords) and a manifest. Then backups/<stack>/restore-<new slot> points at it.
4. launch: cloudformation deploy turning the other slot on, its AMI the latest of the stack's Ami, the old slot's pinned to its own, and --env as ControlEnv if given.
5. boot: waits for the new instance's boot script, which restores the backup before it starts the scheduler and the UI.
6. check: on the new instance, over SSM: restored from this backup, the same pipelines, runs and last run, a schema version no older, every pipeline still paused, the containers running, the UI and each app answering, /system samples in the last 10 minutes, orca status, and the old instance's settings all there. --smoke P=KEY also unpauses P, starts a run of KEY, waits for it to succeed and pauses P again.
7. switch: cloudformation deploy with ActiveControl the new slot, which moves the Elastic IP; it checks the address is on the new instance, stops the old one and fetches the UI from outside.
8. retire: once --keep-old (24h; 0 at once) has passed, --resume turns the old slot off, which terminates its instance.

--rollback before the switch turns the new slot off and starts the old scheduler and UI again. After it, it does the replacement backwards: the new instance quiet and frozen, a backup of its database restored on the old one (started again), the Elastic IP moved back, and the new slot off. --plan prints the steps for this stack without doing any. The pipelines stay paused throughout: unpause them once it's done.