# Maintain and release > Run morca without an assistant: the pull-request routine and the known flaky tests, changelog fragments and database migrations, releasing morca and dista, operating morca and the census in production, and where everything lives. A runbook for the person who owns morca: the pull-request routine, releasing morca and Dista, operating morca and the census, and where things are. Releasing the libraries underneath and the `shelf` CLI is magmalake's runbook, [Maintain and release](https://magmalake.dev/getting-started/maintain). Every command was checked against the source on 2026-11-06 (shelf 0.7.0); where something is a known caveat, it says so. For laying out the checkouts and running the tests, see [Develop locally](https://morca.dev/getting-started/develop). ## Day-to-day pull requests `main` is protected in every morca-dev repository: changes arrive by pull request only, with no direct push and no force push. Cut a branch from a _pushed_ main. A squash-merge folds local commits that were never pushed into your PR under your subject, so check first: ``` git checkout main && git pull --ff-only git log origin/main..main # must print nothing git checkout -b my-change ``` Open the PR with `gh pr create` and watch it with `gh pr checks --watch`. Each repository's `ci.yml` sorts a PR before it spends machine time: a docs-only PR (prose and similar files that no test, build or check reads) skips the code jobs, and a release-only PR (only versions, `CHANGELOG.md`, revs and locks change) takes a short path that skips the nightly leg and the heavy tests. The final `ci` job accepts "skipped" for those, so a green check there is the one to merge on. ### Known flaky tests A red job is real unless it is one of these three, which fail on a loaded runner: - Port collisions in the stand-in EC2 tests: "Address already in use" or "stand-in EC2 didn't start". - `tests/fleet.sh` failing on "by CPU, highest first", when the machine is busy. - `kind` tests that time out inside the vCPU budget. Rerun only the failed jobs, once: ``` gh run list -R morca-dev/morca --branch my-change -L 3 gh run rerun --failed -R morca-dev/morca ``` If it passes, merge. If the same job fails twice, or the message is anything else, it is a real failure: read the log (`gh run view --log-failed`) and fix it. Do not rerun a third time. ### CHANGELOG.md conflicts A repository that has a `changes/` directory next to its CHANGELOG.md keeps a changelog _fragment_ per PR instead of editing the changelog, so two open PRs never conflict there. Add yours with `shelf changes new ` (`--bump minor` for a new feature); it writes `changes/.md`, which you fill in with the entry as it should read. The level is the file name's suffix (`155-fills.minor.md`) or a first line `bump: minor`: `patch`, `minor`, `major` or `breaking`; no level is a patch, and a `### Breaking` or `### Removed` heading makes it breaking. In PR CI, `shelf changes check` fails a PR that changes code without adding or editing a fragment. PRs that change only prose pass, and so do PRs labelled `no-changelog` or `release`. ``` shelf changes new fills --bump minor # writes changes/fills.minor.md shelf changes check # what CI runs; compares HEAD with origin/main ``` Two fragments never conflict, since each PR adds its own file. The repositories that have not moved yet (orca.mojo among them) still use `## Unreleased`: every change adds a line there, so two open PRs conflict. Resolve by keeping both entries. If a release landed in between, the heading is now the version (`## 0.10.0`): put your entry under a fresh `## Unreleased` above it, not under the released version. The level of bump `shelf release` picks comes from these headings, so keep them spelled as the file already does. The design is in [mojoshelf #36](https://github.com/mojoshelf/mojoshelf/issues/36); it needs `shelf` 0.8.0 or later. ### morca's database migrations `_migrations()` in `src/orca/state.mojo` is a list, and the schema version is a migration's _index plus one_, so append only; never insert, edit or reorder. `tests/test_state.mojo` asserts the latest number (30 on 2026-11-06), so a PR that adds a migration bumps that number in the same commit. When two open PRs each add one, they collide in both files: renumber in merge order, the second PR rebasing onto main, moving its migration after the first's and bumping the asserted number by one. A binary refuses a database whose version is newer than its own ("the database is at schema version N, newer than this binary's M"), so deploy the scheduler and UI together. ## Releasing morca and dista `orca-mojo` and `dista-mojo` are tins on mojoshelf, and tins pin each other by git revision, so a release of a library they use is a release of them too. The mechanics, `shelf release plan` and `apply`, the flags, what to do when a release goes wrong and how to release the `shelf` CLI itself, are in magmalake's [Releasing tins with shelf release](https://magmalake.dev/getting-started/maintain#release) and [Releasing the shelf CLI](https://magmalake.dev/getting-started/maintain#cli). Run it from the directory that holds the checkouts, each named after its repository (`orca.mojo`, `dista.mojo`, `iceberg.mojo`, ...), and pull every involved checkout first: the release builds from the local ones. ``` for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do git -C $r pull --ff-only; done shelf release plan orca-mojo shelf release apply --wait --merge # applies .shelf-plan.json ``` After a release, check the registry's newest version and that each released repository's main is green: ``` curl -s https://mojoshelf.org/api/tins/orca-mojo | head -c 600 # newest version, commit_sha for r in morca dista; do echo $r; gh run list -R morca-dev/$r --branch main -L 1; done ``` Then re-pin the consumers. `mojo-census/pixi.toml` pins `dista-mojo`, `parquet-full-mojo`, `iceberg-mojo`, `orca-mojo` and the rest by `rev`; set each to the published commit (a `rev` that was never published builds on your machine and breaks for everyone else), regenerate the lock, and open a PR as in the first section: ``` cd mojo-census && git checkout -b repin $EDITOR pixi.toml # each rev = commit_sha from the registry pixi lock ``` If CI breaks on a sibling after the release, compare every sibling `actions/checkout` `ref:` in `.github/workflows/ci.yml` with the registry's commit for that tin; the workflow pins them separately from `pixi.toml`. ## morca and the census in production ### Where it runs The AWS project was paused twice by its spend limit in November 2026, so the census and morca are moving to one Hetzner Cloud ARM server with a fixed monthly cost ([morca #166](https://github.com/morca-dev/morca/issues/166)). The runbook for it is [docs/hetzner.md](https://github.com/morca-dev/morca/blob/main/docs/hetzner.md) in orca.mojo; it lands with [PR #167](https://github.com/morca-dev/morca/pull/167), so that link is a 404 until that PR merges. Until then the old AWS stack and [orca aws](https://morca.dev/getting-started/step-by-step#aws) still describe what is running. In short, the design is: - one server (CAX21: 4 ARM vCPU, 8 GB), rescaled CPU and RAM only to a CAX41 for the initial import and any full rebuild, then back; - the `local` runner, with Dista threaded workers (`DISTA_WORKER_SLOTS`), so morca launches no instances and spend cannot run away; - the warehouse and logs on Hetzner Object Storage (S3-compatible, through `AWS_ENDPOINT_URL_S3`), Postgres in a container on a Volume with a nightly `pg_dump` to Object Storage; - no open ports but SSH: a Cloudflare Tunnel serves orca.mojourney.dev and mojourney.dev; secrets come from files through `ORCA_SECRET_FETCH`. ### Cutover checklist From #166; the issue is the source if it has changed since. 1. Prerequisites merged and released: morca #165, iceberg-mojo 0.11.1 (the fix for [#164](https://github.com/morca-dev/morca/issues/164)), then orca-mojo 0.10.0; mojo-census re-pinned to those commits (the release steps); the mojoshelf Worker deployed so shelf 0.7.0 installs ([the CLI steps](https://magmalake.dev/getting-started/maintain#cli)). 2. Accounts and credentials, yours to create: a Hetzner Cloud project and read/write API token (for `hcloud` on the laptop only), an Object Storage bucket with S3 keys, a Cloudflare Tunnel and token with routes for orca.mojourney.dev and mojourney.dev, and a GitHub token for the census's mojo\_repos. 3. `deploy/hetzner/up.sh` on the small server; the two hostnames move to the tunnel. 4. `orca user add ` (it reads the password from stdin), then register the census pipelines from its `orca.toml`. 5. `rescale.sh up`, let the fill run (the pipeline page's Fill line shows progress), then `rescale.sh down`. 6. Check the census console and the row counts per day. 7. Tear down the AWS stack only when AWS support has answered and you have decided what to do with the old S3 data. It may hold duplicate or missing hours (#164), so it is not migrated. ### Census settings for the re-import The census registers from its `orca.toml`. Until iceberg 0.11.1 is pinned everywhere, the fill is limited so that concurrent commits cannot claim the same snapshot (#164). For each pipeline that fills: ``` fills = "all" max_runs = 1 fill_max_runs = 1 # and, in the scheduler's environment (the env block or the env file): # DISTA_WORKER_SLOTS= ``` Check the exact keys and where each sits with `orca config show` (what is registered and where each value came from) and morca's [orca-toml reference](https://github.com/morca-dev/morca/blob/main/docs/orca-toml.md). Change a fill's limits in `orca.toml` and `orca register` again; raise them back only once 0.11.1 is pinned. Register by hand with `orca register --config orca.toml ...`. ### Operating The UI is at [orca.mojourney.dev](https://orca.mojourney.dev): **Ops > Control** for the scheduler host, **Ops > Fleet** for launched instances (empty on the `local` runner, and its cost rules are off there), and the **watchdog** page for the rules, their values and active trips. Operations go through the UI or the remote CLI, which keep audit rows. Log in once, then pass `--remote`: ``` orca login https://orca.mojourney.dev [--user NAME] [--days N] [--label L] orca --remote https://orca.mojourney.dev status orca --remote https://orca.mojourney.dev watchdog status orca --remote https://orca.mojourney.dev watchdog snooze 2h # 30m, 1d; "off" ends it orca --remote https://orca.mojourney.dev pause gharchive orca --remote https://orca.mojourney.dev unpause gharchive orca --remote https://orca.mojourney.dev backfill --preview gharchive 2024-01-01..2024-01-31 orca --remote https://orca.mojourney.dev backfill gharchive 2024-01-01..2024-01-31 orca --remote https://orca.mojourney.dev backfill --rerun gharchive 2024-01-05 # done keys too orca --remote https://orca.mojourney.dev cancel gharchive 2024-01-01..2024-01-31 # or: cancel orca --remote https://orca.mojourney.dev resume # rerun starts from a step ``` Without `--remote` the same commands act on the database named by `ORCA_DATABASE_URL` (or set `ORCA_REMOTE`). `backfill` prints "queued N runs"; keys skipped, and why, go to stderr. `--preview` queues nothing. A rerun of a step that appends is refused unless you add `--allow-append` (it would duplicate rows; see the README's "Reruns that replace a partition"). A mistyped command prints the full usage text. Running `docker exec ... orca ...` on the server directly is for debugging only. Caveat: the pipeline name above is an example; list real ones with `orca status`. ## Where things are | Repository | What it is | | --- | --- | | [morca-dev/morca](https://github.com/morca-dev/morca) | The orchestrator: scheduler, UI, CLI, orca aws. Tin orca-mojo. Database migrations live here. | | [morca-dev/census](https://github.com/morca-dev/census) | The example and production workload: three pipelines and the console. Pins every library by commit in pixi.toml; not a tin. | | [morca-dev/dista](https://github.com/morca-dev/dista) | Distributed tasks and actors. Tin dista-mojo. | | [magmalake](https://magmalake.dev/tins) | The libraries morca is built on (Iceberg, Parquet, the codecs, Postgres, Kubernetes), their site, and the releasing runbook. [Where those are](https://magmalake.dev/getting-started/maintain#where). | | [mojoshelf/mojoshelf](https://github.com/mojoshelf/mojoshelf) | The registry and the shelf CLI. | | [morca-dev/morca.dev](https://github.com/morca-dev/morca.dev) | This site. | Issues are tracked per repository, in each one's Issues tab. There are no priority labels, so read the open issues list as it is, and the `release` label where a repository has one. Pull requests are the other queue: `gh pr list -R morca-dev/morca`.