Guides

Steps in Rust

From 0.27.0 a step can be written in Rust. The worker is a Rust library that streams Arrow batches, and a small Mojo pipeline runs it with morca.rust_step, which commits what comes back to the step's Iceberg table in one snapshot. Use it for work that Rust's crates already do well, such as decoding a file format or talking to a service.

The model

Each Rust worker comes with a proxy pipeline in Mojo. The proxy declares what any pipeline declares: its partitions, its steps, and the dataset the step writes. Its step calls rust_step, which loads the worker library over a C ABI, calls it once, and writes the record batches it streams back the way morca writes any step's output: Parquet files, then one Iceberg commit that a retry recognises.

proxy pipeline (Mojo)                     worker (Rust cdylib)
─────────────────────                     ────────────────────
Step[..., writes=["demo.numbers"]]
  ensure_table(ctx, NUMBERS, Schema.parse(SCHEMA))
  rust_step(ctx, "numbers_worker", t) ──▶ morca_worker_run(ctx, NULL, &out)
     check the stream's schema       ◀──  out: ArrowArrayStream
     pull, align, write Parquet      ◀──  get_next, one batch at a time
     one commit: append_files
                    schemas/numbers.json, read by both
  • The worker is a cdylib built with the morca-worker crate. You implement a Worker trait in safe Rust, and export_worker! generates the four C functions morca calls.
  • The schema is declared once, in an Iceberg schema JSON file that both sides embed. cargo test checks the worker's batches against it, and morca checks the stream's schema against the table's before it writes anything.
  • Batches are pulled. morca asks for the next batch when it's ready for one, so a worker that makes each batch when it's pulled holds one at a time and gets backpressure for free. Each batch crosses into Mojo with one copy.
  • Stopping. The worker reads ctx.stopping() between units of work. morca sets it when stop_when() returns true, by default morca.outdated(): a newer version of the pipeline leads, or the run's scheduler drains. A worker that stops ends its stream early, morca commits what came, and RustRun.stopped says so.
  • Failures. An error, or a panic, which the crate catches at the boundary, fails the attempt with its message, and the run's retries apply. Nothing is committed.
  • Resumes. If an earlier attempt of the step already committed in this run, say the process was killed after the commit, a retry finds that commit and returns it, and the worker doesn't run again. There are no duplicate rows.
  • One process per run. The worker runs in the run's own process, so a Rust abort fails only that attempt. It can use the run's granted cores with its own threads, and its memory counts in the run's cgroup like any other step's.

A walkthrough

This is examples/rust-worker in the morca repository, trimmed of the switches its tests use. The worker makes rows numbers for the partition, each with its square and the partition key, batch_rows at a time. CI runs the full example with pixi run -e rust test-rust-worker.

1. Cargo.toml

Build the library as a cdylib for morca to load, and an rlib too, so cargo test can link it. In the repository the dependency is a path; a project of its own pins morca's commit:

[package]
name = "numbers-worker"
version = "0.1.0"
edition = "2021"

[lib]
# libnumbers_worker.so (.dylib on macOS), which the proxy pipeline loads;
# rlib too, for cargo test.
crate-type = ["cdylib", "rlib"]

[dependencies]
morca-worker = { git = "https://github.com/morca-dev/morca", rev = "<morca's commit>" }

2. The schema, schemas/numbers.json

{
  "type": "struct",
  "schema-id": 0,
  "fields": [
    {"id": 1, "name": "id", "required": true, "type": "long"},
    {"id": 2, "name": "partition", "required": true, "type": "string"},
    {"id": 3, "name": "square", "required": false, "type": "double"}
  ]
}

3. The worker, src/lib.rs

Worker::run is called once per attempt and returns an Output, a stream of record batches. Here the stream is an iterator that makes each batch when morca pulls it, and checks the stop flag first.

use std::sync::Arc;

use morca_worker::arrow_array::{ArrayRef, Float64Array, Int64Array, RecordBatch, StringArray};
use morca_worker::{export_worker, schema, Input, Output, Result, RunContext, Worker};

/// The table's schema, the file numbers.mojo embeds too.
pub const SCHEMA: &str = include_str!("../schemas/numbers.json");

pub struct Numbers;

fn int_param(ctx: &RunContext, name: &str, default: i64) -> i64 {
    ctx.params().get(name).and_then(|v| v.as_i64()).unwrap_or(default)
}

impl Worker for Numbers {
    const NAME: &'static str = "numbers";
    const VERSION: &'static str = env!("CARGO_PKG_VERSION");

    fn run(ctx: &RunContext, _input: Option<Input>) -> Result<Output> {
        let rows = int_param(ctx, "rows", 1000);
        let batch_rows = int_param(ctx, "batch_rows", 100).max(1);
        let schema = Arc::new(schema::to_arrow(SCHEMA)?);
        ctx.log(format!("numbers worker {}: {rows} rows for {}", Self::VERSION, ctx.partition()));
        let ctx = ctx.clone();
        let out = schema.clone();
        let mut from = 0;
        // Each batch is made when morca pulls it, so one is in memory at a time.
        let batches = std::iter::from_fn(move || {
            if from >= rows {
                return None;
            }
            if ctx.stopping() {
                ctx.log(format!("stopping after {from} rows, as morca asks"));
                return None;
            }
            let n = batch_rows.min(rows - from);
            let ids: Vec<i64> = (from..from + n).collect();
            let cols: Vec<ArrayRef> = vec![
                Arc::new(Int64Array::from(ids.clone())),
                Arc::new(StringArray::from(vec![ctx.partition(); ids.len()])),
                Arc::new(Float64Array::from_iter_values(ids.iter().map(|&i| (i * i) as f64))),
            ];
            from += n;
            ctx.progress(from as u64, rows as u64);
            Some(RecordBatch::try_new(out.clone(), cols).map_err(Into::into))
        });
        Ok(Output::from_iter(schema, batches))
    }
}

export_worker!(Numbers);

#[cfg(test)]
mod tests {
    use super::*;
    use morca_worker::testing::Run;

    #[test]
    fn makes_its_rows() {
        let ran = Run::new("2024-01")
            .param("rows", 250)
            .param("batch_rows", 100)
            .go::<Numbers>()
            .unwrap();
        assert_eq!(ran.rows(), 250);
        assert_eq!(ran.batches.len(), 3);
        schema::assert_fits(&ran.schema, SCHEMA); // drift from the file fails here
    }
}

4. The proxy pipeline, numbers.mojo

The step creates the table from the same schema file and hands it to rust_step. params is passed to the worker as a JSON object. morca.rust_step isn't re-exported from morca, since it imports iceberg.

from std.os import getenv

from iceberg.schema import Schema
from morca import Ctx, Mapped, Month, Monthly, Pipeline, Step, morca_main
from morca.rust_step import rust_step
from morca.tables import ensure_table

from numbers_schema import NUMBERS_SCHEMA  # schemas/numbers.json, embedded at build time

comptime NUMBERS = "demo.numbers"


def numbers(mut ctx: Ctx, m: Month) raises -> Int:
    var t = ensure_table(ctx, NUMBERS, Schema.parse(NUMBERS_SCHEMA))
    var r = rust_step(
        ctx,
        getenv("NUMBERS_LIB", "numbers_worker"),
        t,
        params=getenv("NUMBERS_PARAMS", "{}"),
    )
    ctx.log(
        String(
            "numbers: ", r.rows, " rows, snapshot ", r.snapshot_id,
            ", stopped=", r.stopped, ", resumed=", r.resumed,
        )
    )
    return r.rows


comptime Numbers = Step[
    "numbers", Month, Int, numbers, Mapped, writes=[NUMBERS]
]
comptime NumbersPipeline = Pipeline[
    "numbers", Numbers, partitions=Monthly["2024-01"]
]


def main() raises:
    morca_main[NumbersPipeline]()

5. Building

Cargo is needed only to build the worker. In the morca repository the rust pixi environment has it. scripts/embed-json.sh writes a Mojo module holding the schema file as a string, so the binary needs no file next to it:

pixi shell -e rust                     # the data environment plus conda-forge's rust
export CARGO_TARGET_DIR=$PWD/build/cargo
cargo test --manifest-path examples/rust-worker/Cargo.toml
cargo build --release --manifest-path examples/rust-worker/Cargo.toml

mkdir -p build/rust-worker
scripts/embed-json.sh examples/rust-worker/schemas/numbers.json NUMBERS_SCHEMA \
    > build/rust-worker/numbers_schema.mojo
scripts/mojo-build.sh -I src -I build/rust-worker $DEP_INC \
    examples/rust-worker/numbers.mojo -o build/numbers

6. Running

export MORCA_WAREHOUSE=$PWD/build/rust-worker/wh
export NUMBERS_LIB=$PWD/build/cargo/release/libnumbers_worker.so   # .dylib on macOS
./build/numbers run --partitions 2024-01

The step's log has the worker's own lines, its progress and what was committed:

numbers worker 0.1.0: 1000 rows for 2024-01
progress 1000/1000
numbers: 1000 rows, snapshot <id>, stopped=False, resumed=False

NUMBERS_LIB is how this example names its library. Without a path, rust_step looks a bare name x up as libx.so (libx.dylib on macOS) in $CONDA_PREFIX/lib, the executable's directory and its ../lib, then the dynamic linker's path.

7. Testing

morca_worker::testing::Run drives a worker through the same C ABI in a cargo test and returns its batches and log lines, as in the test above. The repository's tests/rust_worker.sh also checks, end to end, that a panic and a schema that doesn't fit fail the attempt and commit nothing, that the stop flag ends the stream early with what came committed, and that a run killed after its commit resumes without running the worker again.

What rust_step does

def rust_step(
    mut ctx: Ctx,
    library: String,              # a path, libx.so, or a bare name x
    mut table: Table,
    params: String = "{}",        # a JSON object, the worker's ctx.params()
    step_version: Int = 0,
    stop_when: def() thin -> Bool = outdated,
    flush_bytes: Int = 64 MiB,    # Parquet files of about this size
) raises -> RustRun               # snapshot_id, rows, files, stopped, resumed
  1. If an earlier attempt committed to the table in this run, it returns that commit, with resumed true.
  2. It loads the library when the step runs, never at program start, and refuses one whose ABI version isn't 1.
  3. It calls the worker with the run's context: pipeline, step, partition, run, attempt, code hash, step version, params and secrets.
  4. It checks the stream's schema against the table's, naming the column that doesn't fit.
  5. It pulls batches, aligns each to the table, and writes Parquet as they arrive, in files of about flush_bytes.
  6. At the end of the stream it commits once, with append_files, or replacing the partition for a dataset the step declared in replaces=.

A failure before the commit leaves the files it wrote unreferenced, as an interrupted append does.

The schema rule

Columns are matched by name, at every depth. Every column of the table must be in the worker's schema, and no other. A field id, when the worker gives one, must be the table's, and a required column mustn't be nullable. A type fits when it is the table's or one Iceberg widens to it: long takes Int8 to Int64, double takes Float32 or Float64, string Utf8 or LargeUtf8, and timestamps any unit. schema::to_arrow gives the exact types, each field carrying its Iceberg id. The full table is in rust-workers.md.

Writing a worker

  • RunContext has the partition, run, attempt, the registry hash, the step's version=, params() and param(name), and secret(name). log, progress and metric write to the step's log.
  • Output comes from a reader, a list of batches or an iterator (Output::from_reader, from_batches, from_iter). Work can happen in run, in the stream, or both.
  • Threads are the worker's own: rayon, tokio or plain threads, sized to the run's grant. RunContext is Send + Sync, so a clone can log and check stopping() from another thread. Join every thread before the stream ends.
  • Don't build with panic = "abort": a panic would then take the whole run's process down instead of failing the attempt.
  • Another language can implement the four C functions itself. The header, and the lifetimes and threading rules, are in rust-workers.md.

Shipping it

The runtime image carries the built library, not the Rust toolchain. Make the worker a pixi package whose recipe runs cargo build --release and installs lib/lib<name>.so into the environment, where rust_step finds it like morca's other native libraries. Build it against the morca-worker crate of the morca the pipeline uses: the ABI version is checked when the step runs. Rust's standard library is linked in statically, so the library needs only libc, which suits the distroless runtime image.

Limits

This is ABI version 1 (#422). Not in it yet:

  • A single process. A Rust step runs once, in the run's process. It can't be a Mapped step over dista tasks, spread over a pool's machines with workers = N, or show per-task bars on the run's timeline.
  • No input. The input stream is NULL in version 1; a step's input comes later, without a new ABI version.
  • One output. A worker streams to one table.
  • Parquet files written by the worker, which morca checks and commits through the same path, are coming in a later release.
  • The proxy names the library; there is no rust = "crates/x" key in morca.toml yet.

Hindcast is a real worker: it fetches and decodes NOAA's GFS forecasts with crates.io crates, and Mojo steps score them. docs/rust-workers.md is the full spec.