Getting started

Maintain and release

A runbook for the person who owns morca: the pull-request routine, releasing morca and Dista, operating morca and the census, and where things are. Releasing the libraries underneath and the shelf CLI is magmalake's runbook, Maintain and release. Every command was checked against the source on 2026-11-06 (shelf 0.7.0); where something is a known caveat, it says so. For laying out the checkouts and running the tests, see Develop locally.

Day-to-day pull requests

main is protected in every morca-dev repository: changes arrive by pull request only, with no direct push and no force push. Cut a branch from a pushed main. A squash-merge folds local commits that were never pushed into your PR under your subject, so check first:

git checkout main && git pull --ff-only
git log origin/main..main          # must print nothing
git checkout -b my-change

Open the PR with gh pr create and watch it with gh pr checks --watch. Each repository's ci.yml sorts a PR before it spends machine time: a docs-only PR (prose and similar files that no test, build or check reads) skips the code jobs, and a release-only PR (only versions, CHANGELOG.md, revs and locks change) takes a short path that skips the nightly leg and the heavy tests. The final ci job accepts "skipped" for those, so a green check there is the one to merge on.

Known flaky tests

A red job is real unless it is one of these three, which fail on a loaded runner:

• Port collisions in the stand-in EC2 tests: "Address already in use" or "stand-in EC2 didn't start".
• tests/fleet.sh failing on "by CPU, highest first", when the machine is busy.
• kind tests that time out inside the vCPU budget.

Rerun only the failed jobs, once:

gh run list -R morca-dev/morca --branch my-change -L 3
gh run rerun <run-id> --failed -R morca-dev/morca

If it passes, merge. If the same job fails twice, or the message is anything else, it is a real failure: read the log (gh run view <run-id> --log-failed) and fix it. Do not rerun a third time.

CHANGELOG.md conflicts

A repository that has a changes/ directory next to its CHANGELOG.md keeps a changelog fragment per PR instead of editing the changelog, so two open PRs never conflict there. Add yours with shelf changes new <slug> (--bump minor for a new feature); it writes changes/<slug>.md, which you fill in with the entry as it should read. The level is the file name's suffix (155-fills.minor.md) or a first line bump: minor: patch, minor, major or breaking; no level is a patch, and a ### Breaking or ### Removed heading makes it breaking. In PR CI, shelf changes check fails a PR that changes code without adding or editing a fragment. PRs that change only prose pass, and so do PRs labelled no-changelog or release.

shelf changes new fills --bump minor     # writes changes/fills.minor.md
shelf changes check                      # what CI runs; compares HEAD with origin/main

Two fragments never conflict, since each PR adds its own file. The repositories that have not moved yet (orca.mojo among them) still use ## Unreleased: every change adds a line there, so two open PRs conflict. Resolve by keeping both entries. If a release landed in between, the heading is now the version (## 0.10.0): put your entry under a fresh ## Unreleased above it, not under the released version. The level of bump shelf release picks comes from these headings, so keep them spelled as the file already does. The design is in mojoshelf #36; it needs shelf 0.8.0 or later.

morca's database migrations

_migrations() in src/orca/state.mojo is a list, and the schema version is a migration's index plus one, so append only; never insert, edit or reorder. tests/test_state.mojo asserts the latest number (30 on 2026-11-06), so a PR that adds a migration bumps that number in the same commit. When two open PRs each add one, they collide in both files: renumber in merge order, the second PR rebasing onto main, moving its migration after the first's and bumping the asserted number by one. A binary refuses a database whose version is newer than its own ("the database is at schema version N, newer than this binary's M"), so deploy the scheduler and UI together.

Releasing morca and dista

orca-mojo and dista-mojo are tins on mojoshelf, and tins pin each other by git revision, so a release of a library they use is a release of them too. The mechanics, shelf release plan and apply, the flags, what to do when a release goes wrong and how to release the shelf CLI itself, are in magmalake's Releasing tins with shelf release and Releasing the shelf CLI. Run it from the directory that holds the checkouts, each named after its repository (orca.mojo, dista.mojo, iceberg.mojo, ...), and pull every involved checkout first: the release builds from the local ones.

for r in dista.mojo parquet.mojo iceberg.mojo orca.mojo; do git -C $r pull --ff-only; done
shelf release plan orca-mojo
shelf release apply --wait --merge                # applies .shelf-plan.json

After a release, check the registry's newest version and that each released repository's main is green:

curl -s https://mojoshelf.org/api/tins/orca-mojo | head -c 600       # newest version, commit_sha
for r in morca dista; do echo $r; gh run list -R morca-dev/$r --branch main -L 1; done

Then re-pin the consumers. mojo-census/pixi.toml pins dista-mojo, parquet-full-mojo, iceberg-mojo, orca-mojo and the rest by rev; set each to the published commit (a rev that was never published builds on your machine and breaks for everyone else), regenerate the lock, and open a PR as in the first section:

cd mojo-census && git checkout -b repin
$EDITOR pixi.toml            # each rev = commit_sha from the registry
pixi lock

If CI breaks on a sibling after the release, compare every sibling actions/checkout ref: in .github/workflows/ci.yml with the registry's commit for that tin; the workflow pins them separately from pixi.toml.

morca and the census in production

Where it runs

The AWS project was paused twice by its spend limit in November 2026, so the census and morca are moving to one Hetzner Cloud ARM server with a fixed monthly cost (morca #166). The runbook for it is docs/hetzner.md in orca.mojo; it lands with PR #167, so that link is a 404 until that PR merges. Until then the old AWS stack and orca aws still describe what is running. In short, the design is:

• one server (CAX21: 4 ARM vCPU, 8 GB), rescaled CPU and RAM only to a CAX41 for the initial import and any full rebuild, then back;
• the local runner, with Dista threaded workers (DISTA_WORKER_SLOTS), so morca launches no instances and spend cannot run away;
• the warehouse and logs on Hetzner Object Storage (S3-compatible, through AWS_ENDPOINT_URL_S3), Postgres in a container on a Volume with a nightly pg_dump to Object Storage;
• no open ports but SSH: a Cloudflare Tunnel serves orca.mojourney.dev and mojourney.dev; secrets come from files through ORCA_SECRET_FETCH.

Cutover checklist

From #166; the issue is the source if it has changed since.

1. Prerequisites merged and released: morca #165, iceberg-mojo 0.11.1 (the fix for #164), then orca-mojo 0.10.0; mojo-census re-pinned to those commits (the release steps); the mojoshelf Worker deployed so shelf 0.7.0 installs (the CLI steps).
2. Accounts and credentials, yours to create: a Hetzner Cloud project and read/write API token (for hcloud on the laptop only), an Object Storage bucket with S3 keys, a Cloudflare Tunnel and token with routes for orca.mojourney.dev and mojourney.dev, and a GitHub token for the census's mojo_repos.
3. deploy/hetzner/up.sh on the small server; the two hostnames move to the tunnel.
4. orca user add <name> (it reads the password from stdin), then register the census pipelines from its orca.toml.
5. rescale.sh up, let the fill run (the pipeline page's Fill line shows progress), then rescale.sh down.
6. Check the census console and the row counts per day.
7. Tear down the AWS stack only when AWS support has answered and you have decided what to do with the old S3 data. It may hold duplicate or missing hours (#164), so it is not migrated.

Census settings for the re-import

The census registers from its orca.toml. Until iceberg 0.11.1 is pinned everywhere, the fill is limited so that concurrent commits cannot claim the same snapshot (#164). For each pipeline that fills:

fills = "all"
max_runs = 1
fill_max_runs = 1
# and, in the scheduler's environment (the env block or the env file):
# DISTA_WORKER_SLOTS=<slots per worker>

Check the exact keys and where each sits with orca config show (what is registered and where each value came from) and morca's orca-toml reference. Change a fill's limits in orca.toml and orca register again; raise them back only once 0.11.1 is pinned. Register by hand with orca register --config orca.toml <pipeline binary> ....

Operating

The UI is at orca.mojourney.dev: Ops > Control for the scheduler host, Ops > Fleet for launched instances (empty on the local runner, and its cost rules are off there), and the watchdog page for the rules, their values and active trips. Operations go through the UI or the remote CLI, which keep audit rows. Log in once, then pass --remote:

orca login https://orca.mojourney.dev [--user NAME] [--days N] [--label L]
orca --remote https://orca.mojourney.dev status
orca --remote https://orca.mojourney.dev watchdog status
orca --remote https://orca.mojourney.dev watchdog snooze <rule> 2h        # 30m, 1d; "off" ends it
orca --remote https://orca.mojourney.dev pause gharchive
orca --remote https://orca.mojourney.dev unpause gharchive
orca --remote https://orca.mojourney.dev backfill --preview gharchive 2024-01-01..2024-01-31
orca --remote https://orca.mojourney.dev backfill gharchive 2024-01-01..2024-01-31
orca --remote https://orca.mojourney.dev backfill --rerun gharchive 2024-01-05   # done keys too
orca --remote https://orca.mojourney.dev cancel gharchive 2024-01-01..2024-01-31  # or: cancel <run>
orca --remote https://orca.mojourney.dev resume <run>                      # rerun <run> <step> starts from a step

Without --remote the same commands act on the database named by ORCA_DATABASE_URL (or set ORCA_REMOTE). backfill prints "queued N runs"; keys skipped, and why, go to stderr. --preview queues nothing. A rerun of a step that appends is refused unless you add --allow-append (it would duplicate rows; see the README's "Reruns that replace a partition"). A mistyped command prints the full usage text. Running docker exec ... orca ... on the server directly is for debugging only.

Caveat: the pipeline name above is an example; list real ones with orca status.

Where things are

Issues are tracked per repository, in each one's Issues tab. There are no priority labels, so read the open issues list as it is, and the release label where a repository has one. Pull requests are the other queue: gh pr list -R morca-dev/morca.