Guides
Steps in Rust
From 0.27.0 a step can be written in Rust. The worker is a Rust library that streams
Arrow batches, and a small Mojo pipeline runs it with morca.rust_step,
which commits what comes back to the step's Iceberg table in one snapshot. Use it for
work that Rust's crates already do well, such as decoding a file format or talking
to a service.
The model
Each Rust worker comes with a proxy pipeline in Mojo. The proxy
declares what any pipeline declares: its partitions, its steps, and the dataset the
step writes. Its step calls rust_step, which loads the worker library
over a C ABI, calls it once, and writes the record batches it streams back the way
morca writes any step's output: Parquet files, then one Iceberg commit that a retry
recognises.
proxy pipeline (Mojo) worker (Rust cdylib)
───────────────────── ────────────────────
Step[..., writes=["demo.numbers"]]
ensure_table(ctx, NUMBERS, Schema.parse(SCHEMA))
rust_step(ctx, "numbers_worker", t) ──▶ morca_worker_run(ctx, NULL, &out)
check the stream's schema ◀── out: ArrowArrayStream
pull, align, write Parquet ◀── get_next, one batch at a time
one commit: append_files
schemas/numbers.json, read by both - The worker is a
cdylibbuilt with themorca-workercrate. You implement aWorkertrait in safe Rust, andexport_worker!generates the four C functions morca calls. - The schema is declared once, in an Iceberg schema JSON file that
both sides embed.
cargo testchecks the worker's batches against it, and morca checks the stream's schema against the table's before it writes anything. - Batches are pulled. morca asks for the next batch when it's ready for one, so a worker that makes each batch when it's pulled holds one at a time and gets backpressure for free. Each batch crosses into Mojo with one copy.
- Stopping. The worker reads
ctx.stopping()between units of work. morca sets it whenstop_when()returns true, by defaultmorca.outdated(): a newer version of the pipeline leads, or the run's scheduler drains. A worker that stops ends its stream early, morca commits what came, andRustRun.stoppedsays so. - Failures. An error, or a panic, which the crate catches at the boundary, fails the attempt with its message, and the run's retries apply. Nothing is committed.
- Resumes. If an earlier attempt of the step already committed in this run, say the process was killed after the commit, a retry finds that commit and returns it, and the worker doesn't run again. There are no duplicate rows.
- One process per run. The worker runs in the run's own process, so a Rust abort fails only that attempt. It can use the run's granted cores with its own threads, and its memory counts in the run's cgroup like any other step's.
A walkthrough
This is examples/rust-worker in the morca
repository, trimmed of the switches its tests use. The worker makes
rows numbers for the partition, each with its square and the partition
key, batch_rows at a time. CI runs the full example with
pixi run -e rust test-rust-worker.
1. Cargo.toml
Build the library as a cdylib for morca to load, and an
rlib too, so cargo test can link it. In the repository the
dependency is a path; a project of its own pins morca's commit:
[package]
name = "numbers-worker"
version = "0.1.0"
edition = "2021"
[lib]
# libnumbers_worker.so (.dylib on macOS), which the proxy pipeline loads;
# rlib too, for cargo test.
crate-type = ["cdylib", "rlib"]
[dependencies]
morca-worker = { git = "https://github.com/morca-dev/morca", rev = "<morca's commit>" } 2. The schema, schemas/numbers.json
{
"type": "struct",
"schema-id": 0,
"fields": [
{"id": 1, "name": "id", "required": true, "type": "long"},
{"id": 2, "name": "partition", "required": true, "type": "string"},
{"id": 3, "name": "square", "required": false, "type": "double"}
]
} 3. The worker, src/lib.rs
Worker::run is called once per attempt and returns an
Output, a stream of record batches. Here the stream is an iterator that
makes each batch when morca pulls it, and checks the stop flag first.
use std::sync::Arc;
use morca_worker::arrow_array::{ArrayRef, Float64Array, Int64Array, RecordBatch, StringArray};
use morca_worker::{export_worker, schema, Input, Output, Result, RunContext, Worker};
/// The table's schema, the file numbers.mojo embeds too.
pub const SCHEMA: &str = include_str!("../schemas/numbers.json");
pub struct Numbers;
fn int_param(ctx: &RunContext, name: &str, default: i64) -> i64 {
ctx.params().get(name).and_then(|v| v.as_i64()).unwrap_or(default)
}
impl Worker for Numbers {
const NAME: &'static str = "numbers";
const VERSION: &'static str = env!("CARGO_PKG_VERSION");
fn run(ctx: &RunContext, _input: Option<Input>) -> Result<Output> {
let rows = int_param(ctx, "rows", 1000);
let batch_rows = int_param(ctx, "batch_rows", 100).max(1);
let schema = Arc::new(schema::to_arrow(SCHEMA)?);
ctx.log(format!("numbers worker {}: {rows} rows for {}", Self::VERSION, ctx.partition()));
let ctx = ctx.clone();
let out = schema.clone();
let mut from = 0;
// Each batch is made when morca pulls it, so one is in memory at a time.
let batches = std::iter::from_fn(move || {
if from >= rows {
return None;
}
if ctx.stopping() {
ctx.log(format!("stopping after {from} rows, as morca asks"));
return None;
}
let n = batch_rows.min(rows - from);
let ids: Vec<i64> = (from..from + n).collect();
let cols: Vec<ArrayRef> = vec![
Arc::new(Int64Array::from(ids.clone())),
Arc::new(StringArray::from(vec![ctx.partition(); ids.len()])),
Arc::new(Float64Array::from_iter_values(ids.iter().map(|&i| (i * i) as f64))),
];
from += n;
ctx.progress(from as u64, rows as u64);
Some(RecordBatch::try_new(out.clone(), cols).map_err(Into::into))
});
Ok(Output::from_iter(schema, batches))
}
}
export_worker!(Numbers);
#[cfg(test)]
mod tests {
use super::*;
use morca_worker::testing::Run;
#[test]
fn makes_its_rows() {
let ran = Run::new("2024-01")
.param("rows", 250)
.param("batch_rows", 100)
.go::<Numbers>()
.unwrap();
assert_eq!(ran.rows(), 250);
assert_eq!(ran.batches.len(), 3);
schema::assert_fits(&ran.schema, SCHEMA); // drift from the file fails here
}
} 4. The proxy pipeline, numbers.mojo
The step creates the table from the same schema file and hands it to
rust_step. params is passed to the worker as a JSON object.
morca.rust_step isn't re-exported from morca, since it
imports iceberg.
from std.os import getenv
from iceberg.schema import Schema
from morca import Ctx, Mapped, Month, Monthly, Pipeline, Step, morca_main
from morca.rust_step import rust_step
from morca.tables import ensure_table
from numbers_schema import NUMBERS_SCHEMA # schemas/numbers.json, embedded at build time
comptime NUMBERS = "demo.numbers"
def numbers(mut ctx: Ctx, m: Month) raises -> Int:
var t = ensure_table(ctx, NUMBERS, Schema.parse(NUMBERS_SCHEMA))
var r = rust_step(
ctx,
getenv("NUMBERS_LIB", "numbers_worker"),
t,
params=getenv("NUMBERS_PARAMS", "{}"),
)
ctx.log(
String(
"numbers: ", r.rows, " rows, snapshot ", r.snapshot_id,
", stopped=", r.stopped, ", resumed=", r.resumed,
)
)
return r.rows
comptime Numbers = Step[
"numbers", Month, Int, numbers, Mapped, writes=[NUMBERS]
]
comptime NumbersPipeline = Pipeline[
"numbers", Numbers, partitions=Monthly["2024-01"]
]
def main() raises:
morca_main[NumbersPipeline]() 5. Building
Cargo is needed only to build the worker. In the morca repository the
rust pixi environment has it. scripts/embed-json.sh writes a
Mojo module holding the schema file as a string, so the binary needs no file next to
it:
pixi shell -e rust # the data environment plus conda-forge's rust
export CARGO_TARGET_DIR=$PWD/build/cargo
cargo test --manifest-path examples/rust-worker/Cargo.toml
cargo build --release --manifest-path examples/rust-worker/Cargo.toml
mkdir -p build/rust-worker
scripts/embed-json.sh examples/rust-worker/schemas/numbers.json NUMBERS_SCHEMA \
> build/rust-worker/numbers_schema.mojo
scripts/mojo-build.sh -I src -I build/rust-worker $DEP_INC \
examples/rust-worker/numbers.mojo -o build/numbers 6. Running
export MORCA_WAREHOUSE=$PWD/build/rust-worker/wh
export NUMBERS_LIB=$PWD/build/cargo/release/libnumbers_worker.so # .dylib on macOS
./build/numbers run --partitions 2024-01 The step's log has the worker's own lines, its progress and what was committed:
numbers worker 0.1.0: 1000 rows for 2024-01
progress 1000/1000
numbers: 1000 rows, snapshot <id>, stopped=False, resumed=False NUMBERS_LIB is how this example names its library. Without a path,
rust_step looks a bare name x up as libx.so
(libx.dylib on macOS) in $CONDA_PREFIX/lib, the
executable's directory and its ../lib, then the dynamic linker's path.
7. Testing
morca_worker::testing::Run drives a worker through the same C ABI in a
cargo test and returns its batches and log lines, as in the test above.
The repository's tests/rust_worker.sh also checks, end to end, that a
panic and a schema that doesn't fit fail the attempt and commit nothing, that the
stop flag ends the stream early with what came committed, and that a run killed
after its commit resumes without running the worker again.
What rust_step does
def rust_step(
mut ctx: Ctx,
library: String, # a path, libx.so, or a bare name x
mut table: Table,
params: String = "{}", # a JSON object, the worker's ctx.params()
step_version: Int = 0,
stop_when: def() thin -> Bool = outdated,
flush_bytes: Int = 64 MiB, # Parquet files of about this size
) raises -> RustRun # snapshot_id, rows, files, stopped, resumed - If an earlier attempt committed to the table in this run, it returns that commit, with
resumedtrue. - It loads the library when the step runs, never at program start, and refuses one whose ABI version isn't 1.
- It calls the worker with the run's context: pipeline, step, partition, run, attempt, code hash, step version, params and secrets.
- It checks the stream's schema against the table's, naming the column that doesn't fit.
- It pulls batches, aligns each to the table, and writes Parquet as they arrive, in files of about
flush_bytes. - At the end of the stream it commits once, with
append_files, or replacing the partition for a dataset the step declared inreplaces=.
A failure before the commit leaves the files it wrote unreferenced, as an interrupted
append does.
The schema rule
Columns are matched by name, at every depth. Every column of the table must be in
the worker's schema, and no other. A field id, when the worker gives one, must be the
table's, and a required column mustn't be nullable. A type fits when it is the
table's or one Iceberg widens to it: long takes Int8 to Int64,
double takes Float32 or Float64, string Utf8 or LargeUtf8,
and timestamps any unit. schema::to_arrow gives the exact types, each
field carrying its Iceberg id. The full table is in
rust-workers.md.
Writing a worker
-
RunContexthas the partition, run, attempt, the registry hash, the step'sversion=,params()andparam(name), andsecret(name).log,progressandmetricwrite to the step's log. - Output comes from a reader, a list of batches or an iterator
(
Output::from_reader,from_batches,from_iter). Work can happen inrun, in the stream, or both. - Threads are the worker's own: rayon, tokio or plain threads, sized
to the run's grant.
RunContextisSend + Sync, so a clone can log and checkstopping()from another thread. Join every thread before the stream ends. - Don't build with
panic = "abort": a panic would then take the whole run's process down instead of failing the attempt. - Another language can implement the four C functions itself. The header, and the lifetimes and threading rules, are in rust-workers.md.
Shipping it
The runtime image carries the built library, not the Rust toolchain. Make the worker
a pixi package whose recipe runs cargo build --release and installs
lib/lib<name>.so into the environment, where
rust_step finds it like morca's other native libraries. Build it against
the morca-worker crate of the morca the pipeline uses: the ABI version is
checked when the step runs. Rust's standard library is linked in statically, so the
library needs only libc, which suits the
distroless runtime image.
Limits
This is ABI version 1 (#422). Not in it yet:
- A single process. A Rust step runs once, in the run's process. It
can't be a
Mappedstep over dista tasks, spread over a pool's machines withworkers = N, or show per-task bars on the run's timeline. - No input. The input stream is NULL in version 1; a step's input comes later, without a new ABI version.
- One output. A worker streams to one table.
- Parquet files written by the worker, which morca checks and commits through the same path, are coming in a later release.
- The proxy names the library; there is no
rust = "crates/x"key in morca.toml yet.
Hindcast is a real worker: it fetches and decodes NOAA's GFS forecasts with crates.io crates, and Mojo steps score them. docs/rust-workers.md is the full spec.