# Steps in Rust

> Writing a morca step in Rust (from 0.27.0): a worker library built with the morca-worker crate, loaded over a C ABI by a Mojo proxy pipeline that calls morca.rust_step, Arrow batches streamed back and committed in one snapshot, one schema file read by both sides, stopping, panics and resumes, a complete walkthrough, and the limits of ABI version 1.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

From 0.27.0 a step can be written in Rust. The worker is a Rust library that streams Arrow batches, and a small Mojo pipeline runs it with `morca.rust_step`, which commits what comes back to the step's Iceberg table in one snapshot. Use it for work that Rust's crates already do well, such as decoding a file format or talking to a service.

## The model

Each Rust worker comes with a **proxy pipeline in Mojo**. The proxy declares what any pipeline declares: its partitions, its steps, and the dataset the step writes. Its step calls `rust_step`, which loads the worker library over a C ABI, calls it once, and writes the record batches it streams back the way morca writes any step's output: Parquet files, then one Iceberg commit that a retry recognises.

```
proxy pipeline (Mojo)                     worker (Rust cdylib)
─────────────────────                     ────────────────────
Step[..., writes=["demo.numbers"]]
  ensure_table(ctx, NUMBERS, Schema.parse(SCHEMA))
  rust_step(ctx, "numbers_worker", t) ──▶ morca_worker_run(ctx, NULL, &out)
     check the stream's schema       ◀──  out: ArrowArrayStream
     pull, align, write Parquet      ◀──  get_next, one batch at a time
     one commit: append_files
                    schemas/numbers.json, read by both
```

- **The worker** is a `cdylib` built with the [morca-worker](https://github.com/morca-dev/morca/tree/main/rust/morca-worker) crate. You implement a `Worker` trait in safe Rust, and `export_worker!` generates the four C functions morca calls.
- **The schema is declared once**, in an Iceberg schema JSON file that both sides embed. `cargo test` checks the worker's batches against it, and morca checks the stream's schema against the table's before it writes anything.
- **Batches are pulled.** morca asks for the next batch when it's ready for one, so a worker that makes each batch when it's pulled holds one at a time and gets backpressure for free. Each batch crosses into Mojo with one copy.
- **Stopping.** The worker reads `ctx.stopping()` between units of work. morca sets it when `stop_when()` returns true, by default `morca.outdated()`: a newer version of the pipeline leads, or the run's scheduler drains. A worker that stops ends its stream early, morca commits what came, and `RustRun.stopped` says so.
- **Failures.** An error, or a panic, which the crate catches at the boundary, fails the attempt with its message, and the run's retries apply. Nothing is committed.
- **Resumes.** If an earlier attempt of the step already committed in this run, say the process was killed after the commit, a retry finds that commit and returns it, and the worker doesn't run again. There are no duplicate rows.
- **One process per run.** The worker runs in the run's own process, so a Rust abort fails only that attempt. It can use the run's granted cores with its own threads, and its memory counts in the run's cgroup like any other step's.

## A walkthrough

This is [examples/rust-worker](https://github.com/morca-dev/morca/tree/main/examples/rust-worker) in the morca repository, trimmed of the switches its tests use. The worker makes `rows` numbers for the partition, each with its square and the partition key, `batch_rows` at a time. CI runs the full example with `pixi run -e rust test-rust-worker`.

### 1\. Cargo.toml

Build the library as a `cdylib` for morca to load, and an `rlib` too, so `cargo test` can link it. In the repository the dependency is a path; a project of its own pins morca's commit:

```
[package]
name = "numbers-worker"
version = "0.1.0"
edition = "2021"

[lib]
# libnumbers_worker.so (.dylib on macOS), which the proxy pipeline loads;
# rlib too, for cargo test.
crate-type = ["cdylib", "rlib"]

[dependencies]
morca-worker = { git = "https://github.com/morca-dev/morca", rev = "<morca's commit>" }
```

### 2\. The schema, schemas/numbers.json

```
{
  "type": "struct",
  "schema-id": 0,
  "fields": [
    {"id": 1, "name": "id", "required": true, "type": "long"},
    {"id": 2, "name": "partition", "required": true, "type": "string"},
    {"id": 3, "name": "square", "required": false, "type": "double"}
  ]
}
```

### 3\. The worker, src/lib.rs

`Worker::run` is called once per attempt and returns an `Output`, a stream of record batches. Here the stream is an iterator that makes each batch when morca pulls it, and checks the stop flag first.

```
use std::sync::Arc;

use morca_worker::arrow_array::{ArrayRef, Float64Array, Int64Array, RecordBatch, StringArray};
use morca_worker::{export_worker, schema, Input, Output, Result, RunContext, Worker};

/// The table's schema, the file numbers.mojo embeds too.
pub const SCHEMA: &str = include_str!("../schemas/numbers.json");

pub struct Numbers;

fn int_param(ctx: &RunContext, name: &str, default: i64) -> i64 {
    ctx.params().get(name).and_then(|v| v.as_i64()).unwrap_or(default)
}

impl Worker for Numbers {
    const NAME: &'static str = "numbers";
    const VERSION: &'static str = env!("CARGO_PKG_VERSION");

    fn run(ctx: &RunContext, _input: Option<Input>) -> Result<Output> {
        let rows = int_param(ctx, "rows", 1000);
        let batch_rows = int_param(ctx, "batch_rows", 100).max(1);
        let schema = Arc::new(schema::to_arrow(SCHEMA)?);
        ctx.log(format!("numbers worker {}: {rows} rows for {}", Self::VERSION, ctx.partition()));
        let ctx = ctx.clone();
        let out = schema.clone();
        let mut from = 0;
        // Each batch is made when morca pulls it, so one is in memory at a time.
        let batches = std::iter::from_fn(move || {
            if from >= rows {
                return None;
            }
            if ctx.stopping() {
                ctx.log(format!("stopping after {from} rows, as morca asks"));
                return None;
            }
            let n = batch_rows.min(rows - from);
            let ids: Vec<i64> = (from..from + n).collect();
            let cols: Vec<ArrayRef> = vec![
                Arc::new(Int64Array::from(ids.clone())),
                Arc::new(StringArray::from(vec![ctx.partition(); ids.len()])),
                Arc::new(Float64Array::from_iter_values(ids.iter().map(|&i| (i * i) as f64))),
            ];
            from += n;
            ctx.progress(from as u64, rows as u64);
            Some(RecordBatch::try_new(out.clone(), cols).map_err(Into::into))
        });
        Ok(Output::from_iter(schema, batches))
    }
}

export_worker!(Numbers);

#[cfg(test)]
mod tests {
    use super::*;
    use morca_worker::testing::Run;

    #[test]
    fn makes_its_rows() {
        let ran = Run::new("2024-01")
            .param("rows", 250)
            .param("batch_rows", 100)
            .go::<Numbers>()
            .unwrap();
        assert_eq!(ran.rows(), 250);
        assert_eq!(ran.batches.len(), 3);
        schema::assert_fits(&ran.schema, SCHEMA); // drift from the file fails here
    }
}
```

### 4\. The proxy pipeline, numbers.mojo

The step creates the table from the same schema file and hands it to `rust_step`. `params` is passed to the worker as a JSON object. `morca.rust_step` isn't re-exported from `morca`, since it imports iceberg.

```
from std.os import getenv

from iceberg.schema import Schema
from morca import Ctx, Mapped, Month, Monthly, Pipeline, Step, morca_main
from morca.rust_step import rust_step
from morca.tables import ensure_table

from numbers_schema import NUMBERS_SCHEMA  # schemas/numbers.json, embedded at build time

comptime NUMBERS = "demo.numbers"

def numbers(mut ctx: Ctx, m: Month) raises -> Int:
    var t = ensure_table(ctx, NUMBERS, Schema.parse(NUMBERS_SCHEMA))
    var r = rust_step(
        ctx,
        getenv("NUMBERS_LIB", "numbers_worker"),
        t,
        params=getenv("NUMBERS_PARAMS", "{}"),
    )
    ctx.log(
        String(
            "numbers: ", r.rows, " rows, snapshot ", r.snapshot_id,
            ", stopped=", r.stopped, ", resumed=", r.resumed,
        )
    )
    return r.rows

comptime Numbers = Step[
    "numbers", Month, Int, numbers, Mapped, writes=[NUMBERS]
]
comptime NumbersPipeline = Pipeline[
    "numbers", Numbers, partitions=Monthly["2024-01"]
]

def main() raises:
    morca_main[NumbersPipeline]()
```

### 5\. Building

Cargo is needed only to build the worker. In the morca repository the `rust` pixi environment has it. `scripts/embed-json.sh` writes a Mojo module holding the schema file as a string, so the binary needs no file next to it:

```
pixi shell -e rust                     # the data environment plus conda-forge's rust
export CARGO_TARGET_DIR=$PWD/build/cargo
cargo test --manifest-path examples/rust-worker/Cargo.toml
cargo build --release --manifest-path examples/rust-worker/Cargo.toml

mkdir -p build/rust-worker
scripts/embed-json.sh examples/rust-worker/schemas/numbers.json NUMBERS_SCHEMA \
    > build/rust-worker/numbers_schema.mojo
scripts/mojo-build.sh -I src -I build/rust-worker $DEP_INC \
    examples/rust-worker/numbers.mojo -o build/numbers
```

### 6\. Running

```
export MORCA_WAREHOUSE=$PWD/build/rust-worker/wh
export NUMBERS_LIB=$PWD/build/cargo/release/libnumbers_worker.so   # .dylib on macOS
./build/numbers run --partitions 2024-01
```

The step's log has the worker's own lines, its progress and what was committed:

```
numbers worker 0.1.0: 1000 rows for 2024-01
progress 1000/1000
numbers: 1000 rows, snapshot <id>, stopped=False, resumed=False
```

`NUMBERS_LIB` is how this example names its library. Without a path, `rust_step` looks a bare name `x` up as `libx.so` (`libx.dylib` on macOS) in `$CONDA_PREFIX/lib`, the executable's directory and its `../lib`, then the dynamic linker's path.

### 7\. Testing

`morca_worker::testing::Run` drives a worker through the same C ABI in a `cargo test` and returns its batches and log lines, as in the test above. The repository's `tests/rust_worker.sh` also checks, end to end, that a panic and a schema that doesn't fit fail the attempt and commit nothing, that the stop flag ends the stream early with what came committed, and that a run killed after its commit resumes without running the worker again.

## What rust\_step does

```
def rust_step(
    mut ctx: Ctx,
    library: String,              # a path, libx.so, or a bare name x
    mut table: Table,
    params: String = "{}",        # a JSON object, the worker's ctx.params()
    step_version: Int = 0,
    stop_when: def() thin -> Bool = outdated,
    flush_bytes: Int = 64 MiB,    # Parquet files of about this size
) raises -> RustRun               # snapshot_id, rows, files, stopped, resumed
```

1. If an earlier attempt committed to the table in this run, it returns that commit, with `resumed` true.
2. It loads the library when the step runs, never at program start, and refuses one whose ABI version isn't 1.
3. It calls the worker with the run's context: pipeline, step, partition, run, attempt, code hash, step version, params and secrets.
4. It checks the stream's schema against the table's, naming the column that doesn't fit.
5. It pulls batches, aligns each to the table, and writes Parquet as they arrive, in files of about `flush_bytes`.
6. At the end of the stream it commits once, with `append_files`, or replacing the partition for a dataset the step declared in `replaces=`.

A failure before the commit leaves the files it wrote unreferenced, as an interrupted `append` does.

## The schema rule

Columns are matched by name, at every depth. Every column of the table must be in the worker's schema, and no other. A field id, when the worker gives one, must be the table's, and a required column mustn't be nullable. A type fits when it is the table's or one Iceberg widens to it: `long` takes Int8 to Int64, `double` takes Float32 or Float64, `string` Utf8 or LargeUtf8, and timestamps any unit. `schema::to_arrow` gives the exact types, each field carrying its Iceberg id. The full table is in [rust-workers.md](https://github.com/morca-dev/morca/blob/main/docs/rust-workers.md#the-schema-one-file-checked-twice).

## Writing a worker

- **`RunContext`** has the partition, run, attempt, the registry hash, the step's `version=`, `params()` and `param(name)`, and `secret(name)`. `log`, `progress` and `metric` write to the step's log.
- **Output** comes from a reader, a list of batches or an iterator (`Output::from_reader`, `from_batches`, `from_iter`). Work can happen in `run`, in the stream, or both.
- **Threads** are the worker's own: rayon, tokio or plain threads, sized to the run's grant. `RunContext` is `Send + Sync`, so a clone can log and check `stopping()` from another thread. Join every thread before the stream ends.
- **Don't build with `panic = "abort"`**: a panic would then take the whole run's process down instead of failing the attempt.
- **Another language** can implement the four C functions itself. The header, and the lifetimes and threading rules, are in [rust-workers.md](https://github.com/morca-dev/morca/blob/main/docs/rust-workers.md#the-c-abi).

## Shipping it

The runtime image carries the built library, not the Rust toolchain. Make the worker a pixi package whose recipe runs `cargo build --release` and installs `lib/lib<name>.so` into the environment, where `rust_step` finds it like morca's other native libraries. Build it against the `morca-worker` crate of the morca the pipeline uses: the ABI version is checked when the step runs. Rust's standard library is linked in statically, so the library needs only libc, which suits the [distroless runtime image](https://morca.dev/docs/deploy#runtime).

## Limits

This is ABI version 1 ([#422](https://github.com/morca-dev/morca/pull/422)). Not in it yet:

- **A single process.** A Rust step runs once, in the run's process. It can't be a `Mapped` step over dista tasks, spread over a pool's machines with `workers = N`, or show per-task bars on the run's timeline.
- **No input.** The input stream is NULL in version 1; a step's input comes later, without a new ABI version.
- **One output.** A worker streams to one table.
- **Parquet files written by the worker**, which morca checks and commits through the same path, are coming in a later release.
- The proxy names the library; there is no `rust = "crates/x"` key in morca.toml yet.

[Hindcast](https://morca.dev/docs/hindcast) is a real worker: it fetches and decodes NOAA's GFS forecasts with crates.io crates, and Mojo steps score them. [docs/rust-workers.md](https://github.com/morca-dev/morca/blob/main/docs/rust-workers.md) is the full spec.

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.
