# Hindcast

> Hindcast, morca's showcase for steps in Rust: it scores NOAA GFS forecasts of 2 m temperature, 0 to 7 days ahead, with a Rust worker that Range-fetches and decodes one GRIB2 message per lead and Mojo steps that compute bias, MAE and RMSE per lead and a 10-city scorecard. How incomplete cycles work, how to run it and backfill 30 days, and the numbers from a real run.

[morca](https://morca.dev/)

[](https://morca.dev/search "Search")

How good were NOAA's weather forecasts? Hindcast fetches each day's GFS forecast of 2 m temperature, 0 to 7 days ahead, and scores it against what the next days' analyses say happened. The fetching and decoding are a [Rust worker](https://morca.dev/docs/rust) built from crates.io crates; the scoring is ordinary Mojo steps over Iceberg tables.

## What it does

```
hindcast.mojo, partitions: forecast cycles (00z), Daily from 2021-03-23, 06:00 UTC for the day before

forecasts [mapped, Rust]                      skill [gather, Mojo]
  rust_step ──▶ libhindcast_worker              for each valid day V the run touches:
    per lead f000 … f168:                         analysis = f000 of cycle V
      GET gfs…f024.idx                            for each lead L: forecast of cycle V - L
      GET gfs…f024, Range: one message            bias, MAE, RMSE (cos-latitude weighted)
      decode (grib, template 5.3)                 and the city scorecard
      ──▶ one Arrow batch, 65,160 rows      ──▶ hindcast.skill, hindcast.cities
  ──▶ hindcast.t2m                                (replace V's partition)
      (replaces the cycle's partitions)
```

- **Data.** GFS at 1°, the field `TMP:2 m above ground`, the 00z run of each day, at leads f000, f024 … f168, from NOAA's open bucket (`noaa-gfs-bdp-pds`, no credentials). A file is 45 MB, but the worker reads its `.idx`, finds the one message, and fetches only that with an HTTP Range request: about 74 KB a lead.
- **`forecasts`**, the Rust step. Each lead is decoded with the `grib` crate and becomes one Arrow batch of 65,160 rows (cycle, lead\_hours, valid\_time, lat, lon, t2m\_k), made when morca pulls it, so the worker holds one lead at a time. The rows go into `hindcast.t2m`, partitioned by cycle and lead, and a rerun replaces the cycle's partitions. The library links only libc: `grib` without its C codecs, and `ureq` over rustls with webpki's roots, so there's no OpenSSL and no CA file to find in the distroless runtime image.
- **`skill`**, the Mojo step. A forecast of cycle C at lead L is valid at V = C + L, and is scored against V's analysis, the f000 of cycle V. It computes the grid-wide bias, MAE and RMSE of forecast minus analysis, each point weighted by cos(latitude), into `hindcast.skill`, and the error at the grid point nearest each of ten cities (New York, London, Tokyo, Sydney, São Paulo, Cairo, Mumbai, Moscow, Lagos, Anchorage) into `hindcast.cities`.

The pipeline itself is two steps and a daily schedule:

```
comptime Forecasts = Step[
    "forecasts", Day, Fetched, forecasts, Mapped, retries=2, replaces=[T2M]
]
comptime Skill = Step[
    "skill", Seq[Fetched], Scored, skill, Gather,
    after=Forecasts, reads=[T2M], replaces=[SKILL, CITIES],
]
comptime Hindcast = Pipeline[
    "hindcast", Forecasts, Skill,
    partitions=Daily["2021-03-23"],
    schedule=Cron["0 6 * * *"],
]
```

## Incomplete cycles

A forecast can only be scored once its valid day's analysis is fetched, so a cycle's 7-day forecast waits a week. Hindcast handles that without dependencies between partitions. A run's `skill` step works out the valid days its cycles touch: each cycle's own day, whose analysis scores the forecasts of the seven cycles before, and the seven days after, where its own forecasts are scored if those analyses are already there. For each such day it recomputes everything valid that day from `hindcast.t2m` and replaces the day's partition. A forecast whose analysis isn't there yet is skipped, and scored by the run that fetches it.

So the scores don't depend on the order cycles arrive in: daily runs, a backfill in any order, or a rerun all end with every forecast scored whose two cycles are fetched. A cycle is **complete** once all seven leads are scored, and `hindcast report` lists the cycles that aren't yet.

A cycle that isn't in the bucket yet fails the worker with "not published yet". For a cycle of the last two days the step then defers half an hour with `morca.defer`, spending no retry; for an older one it fails.

## Running it

From the root of the [morca repository](https://github.com/morca-dev/morca), in the `rust` pixi environment:

```
pixi shell -e rust
export CARGO_TARGET_DIR=$PWD/build/cargo
cargo build --release --manifest-path examples/hindcast/Cargo.toml
mkdir -p build/hindcast
scripts/embed-json.sh examples/hindcast/schemas/t2m.json T2M_SCHEMA > build/hindcast/t2m_schema.mojo
scripts/mojo-build.sh -I src -I build/hindcast -I examples/hindcast $DEP_INC \
    examples/hindcast/hindcast.mojo -o build/hindcast/hindcast

export MORCA_WAREHOUSE=$PWD/build/hindcast/wh
export HINDCAST_LIB=$PWD/build/cargo/release/libhindcast_worker.so   # .dylib on macOS
./build/hindcast/hindcast run --partitions 2026-11-01..2026-11-12
./build/hindcast/hindcast report 2026-11-01
```

That run of twelve cycles fetches 96 messages, about 7 MB, and scores what they allow: the first five cycles completely, the later ones as far as 2026-11-12's analysis goes. `hindcast report [cycle]` prints the skill per lead, the cycles still waiting for analyses, the tables' row counts, and with a cycle, its rows and city scorecard. `HINDCAST_SOURCE` points the worker at another base URL or a directory laid out as the bucket, and `HINDCAST_LEADS` picks the leads.

## A 30-day backfill

Register the pipeline with a scheduler, which needs a database (`MORCA_DATABASE_URL`), and backfill the days. The scheduler's runs inherit its environment, so set `MORCA_WAREHOUSE` and `HINDCAST_LIB` where it runs:

```
pixi run morca    # builds ./build/morca
./build/morca register ./build/hindcast/hindcast
./build/morca backfill hindcast 2026-10-01..2026-10-30
./build/morca scheduler
./build/morca status hindcast
./build/hindcast/hindcast report
```

The backfill queues one run per day. Each fetches its cycle with eight Range requests and rescores the valid days it touches, so the scores fill in as runs finish, whatever order they finish in. Cycles to 2026-10-23 are complete when the backfill is done; the last seven wait for November's analyses, which the daily schedule or another backfill brings. `morca backfill --rerun hindcast <days>` fetches and scores days again, replacing what was there.

## Real numbers

A backfill of 13 cycles, 2026-10-31 to 2026-11-12, against the live bucket: all 13 runs succeeded in about 4 s each, with a peak of 44 to 63 MB. The tables ended with 6,776,640 rows in `t2m`, 63 in `skill` and 630 in `cities`; 6 cycles were complete and the last 7 waiting for analyses, as designed ([#423](https://github.com/morca-dev/morca/pull/423)). The skill per lead is the mean bias and MAE over the cycles scored, and the RMSE over all of them:

| Lead | Cycles | Bias (K) | MAE (K) | RMSE (K) |
| --- | --- | --- | --- | --- |
| 24h | 12 | −0.0147 | 0.3685 | 0.6464 |
| 48h | 11 | −0.0297 | 0.5403 | 0.9343 |
| 72h | 10 | −0.0397 | 0.6743 | 1.1684 |
| 96h | 9 | −0.0412 | 0.8028 | 1.3879 |
| 120h | 8 | −0.0563 | 0.9450 | 1.6500 |
| 144h | 7 | −0.0553 | 1.0928 | 1.8927 |
| 168h | 6 | −0.0666 | 1.2441 | 2.1848 |

The error grows steadily with lead, from 0.65 K a day ahead to 2.18 K a week ahead, and the bias stays small and slightly cold. Single cities vary far more: the 2026-11-01 forecast for 168 hours ahead was off by −6.7 K in São Paulo and +2.3 K in New York.

## Tests

`pixi run -e rust test-hindcast` runs offline, in CI, on four real 74 KB messages cut from NOAA's files. It checks the `.idx` lookup and the decode against ecCodes, the library's exports and that it links no Rust std or OpenSSL, that two cycles run in either order give the same scores, the skill and city numbers against ecCodes and numpy, and that an incomplete cycle is reported, a rerun replaces, and a missing cycle defers or fails by its age. The [README](https://github.com/morca-dev/morca/blob/main/examples/hindcast/README.md) and [the code](https://github.com/morca-dev/morca/tree/main/examples/hindcast) have the rest.

[morca](https://morca.dev/)

Data pipelines in Mojo. Early, and built in the open.

### Project

[GitHub organisation](https://github.com/morca-dev) [Issues](https://github.com/morca-dev/morca/issues) [morca](https://github.com/morca-dev/morca) [dista](https://github.com/morca-dev/dista) [census](https://github.com/morca-dev/census)

### Built on

[magmalake.dev](https://magmalake.dev) [The Iceberg, Parquet and codec libraries](https://magmalake.dev/tins) [mojoshelf.org](https://mojoshelf.org) [The Mojo language](https://www.modular.com/mojo)

### Standards

[Apache Iceberg](https://iceberg.apache.org) [Apache Parquet](https://parquet.apache.org) [PostgreSQL](https://www.postgresql.org)

Copyright 2026 Marius Seritan. Every repository is licensed [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

Apache Iceberg and Parquet are trademarks of the Apache Software Foundation. Mojo is a trademark of Modular Inc. This project is not affiliated with or endorsed by the ASF or Modular.
