morca

Docs

Using morca

morca turns a Mojo file of steps into a pipeline binary with a command line, and a scheduler runs those binaries on this machine, on Kubernetes or on EC2, recording every run in Postgres. This page installs it; the pages below cover the rest. The plan is in the repository's DESIGN.md.

The docs

  • Features: Everything morca does on one page, major features first, each linking to where it is explained.
  • Writing a pipeline: Steps and pipelines, partitions by date or by name, schedules, Iceberg tables and datasets, sensors, Dista inside steps, and building the binaries.
  • The command line: The commands you type, on a pipeline binary and on orca, from a laptop with orca login and --remote, the ones orca runs for you, and how the same binary acts locally or on AWS.
  • Scheduling and running: The scheduler, runs on Kubernetes and EC2, placement and remote workers, fills, sizing runs by memory and CPU, the watchdog and alerts, runs in Postgres, logs and machine metrics, lineage and rebuilds, the environment, and what happens when things fail.
  • The UI: The pages and their JSON, coverage, filtering runs and acting on many, the actions and what they ask first, logins, a run's page, and code on hover.
  • Debugging failures: Where a failed run's errors and logs are, adding context to errors, symbolized crash dumps, reproducing a partition locally, and resuming it.
  • Remote workers: Running some pipelines on a second machine: a worker token, a placement, a worker container that needs only outbound HTTPS, and the census as the example.
  • Mojo API: The Mojo API of the orca package, module by module, from the published release.
  • orca.toml: Every key of orca.toml: pipelines, pools, fills, requests, how many runs at once, and what register refuses.
  • Files and HTTP API: The HTTP and JSON API, the worker API, logs and metrics files, the Postgres tables and the environment variables.
  • Deploying: Deploying on one Hetzner server: the scripts from a release, deploys that replace only what changed and don't pause runs, rollbacks, settings and SOPS secrets, and rescaling.
  • Pools and autoscaling: Worker pools whose machines morca makes and removes on Hetzner Cloud or EC2, and one run spread over several machines.
  • Sizing and scaling: The right size for each run and the right number of runs: requests, learned requests, auto counts and memory pressure.
  • Alerts and the watchdog: Alerts, the watchdog, the metrics morca samples, and where to look when a run is stuck.
  • Upgrades and rollbacks: New releases, additive migrations, rolling back without restoring the database, and backups.
  • Tokens and secrets: Logins, worker and machine tokens, each run's cluster token, pipeline secrets and cloud API tokens.
  • Roadmap: Vertical scaling, the right size for each run, and horizontal scaling, the right number of runs and machines: what morca does today, what's next, and the work it draws on.

Install it

morca is the mojoshelf tin orca-mojo. With pixi and mojoshelf's pixi shelf installed, a new project adds it, and the tins it needs come with it. Plain pixi can add it from its repository instead:

mojoshelf:

pixi init -c https://conda.modular.com/max-nightly -c conda-forge lake && cd lake
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml   # tins build with pixi-build
pixi add "mojo-compiler==1.2.0"
pixi shelf add orca-mojo

pixi shelf comes with mojoshelf: install it first.

A pipeline is one Mojo file whose main hands it to orca_main, compiled into a binary of its own. Build one and run it in memory to try it: 

pixi run mojo build -I src pipelines/lake.mojo -o build/lake
./build/lake run --partitions 2024-01..2024-03

Writing a pipeline says what goes in the file, and how to build the orca command that schedules it.

The playground

To see the UI before running anything real, the repository's playground starts it on this machine with a throwaway Postgres, the example pipelines and their runs, two of them on a stand-in EC2: 

git clone https://github.com/morca-dev/morca orca.mojo && cd orca.mojo
git clone https://github.com/morca-dev/dista ../dista.mojo
git clone https://github.com/magmalake/k8s.mojo ../k8s.mojo
pixi run playground          # then http://127.0.0.1:8270, log in as demo / demo-password

From source

Building from source works the same way: the repository compiles Dista and k8s.mojo from those checkouts beside it, pixi run test runs the tests, and pixi run orca builds the scheduler to build/orca. It targets Mojo 1.2.0, with a nightly environment alongside. Its state goes through postgres-mojo, which loads libpq; the tests start a throwaway Postgres server of their own. orca.tables, the Iceberg part, has its tests in the data environment (pixi run -e data test-tables).