Guides
Hindcast
How good were NOAA's weather forecasts? Hindcast fetches each day's GFS forecast of 2 m temperature, 0 to 7 days ahead, and scores it against what the next days' analyses say happened. The fetching and decoding are a Rust worker built from crates.io crates; the scoring is ordinary Mojo steps over Iceberg tables.
What it does
hindcast.mojo, partitions: forecast cycles (00z), Daily from 2021-03-23, 06:00 UTC for the day before
forecasts [mapped, Rust] skill [gather, Mojo]
rust_step ──▶ libhindcast_worker for each valid day V the run touches:
per lead f000 … f168: analysis = f000 of cycle V
GET gfs…f024.idx for each lead L: forecast of cycle V - L
GET gfs…f024, Range: one message bias, MAE, RMSE (cos-latitude weighted)
decode (grib, template 5.3) and the city scorecard
──▶ one Arrow batch, 65,160 rows ──▶ hindcast.skill, hindcast.cities
──▶ hindcast.t2m (replace V's partition)
(replaces the cycle's partitions) - Data. GFS at 1°, the field
TMP:2 m above ground, the 00z run of each day, at leads f000, f024 … f168, from NOAA's open bucket (noaa-gfs-bdp-pds, no credentials). A file is 45 MB, but the worker reads its.idx, finds the one message, and fetches only that with an HTTP Range request: about 74 KB a lead. -
forecasts, the Rust step. Each lead is decoded with thegribcrate and becomes one Arrow batch of 65,160 rows (cycle, lead_hours, valid_time, lat, lon, t2m_k), made when morca pulls it, so the worker holds one lead at a time. The rows go intohindcast.t2m, partitioned by cycle and lead, and a rerun replaces the cycle's partitions. The library links only libc:gribwithout its C codecs, andureqover rustls with webpki's roots, so there's no OpenSSL and no CA file to find in the distroless runtime image. -
skill, the Mojo step. A forecast of cycle C at lead L is valid at V = C + L, and is scored against V's analysis, the f000 of cycle V. It computes the grid-wide bias, MAE and RMSE of forecast minus analysis, each point weighted by cos(latitude), intohindcast.skill, and the error at the grid point nearest each of ten cities (New York, London, Tokyo, Sydney, São Paulo, Cairo, Mumbai, Moscow, Lagos, Anchorage) intohindcast.cities.
The pipeline itself is two steps and a daily schedule:
comptime Forecasts = Step[
"forecasts", Day, Fetched, forecasts, Mapped, retries=2, replaces=[T2M]
]
comptime Skill = Step[
"skill", Seq[Fetched], Scored, skill, Gather,
after=Forecasts, reads=[T2M], replaces=[SKILL, CITIES],
]
comptime Hindcast = Pipeline[
"hindcast", Forecasts, Skill,
partitions=Daily["2021-03-23"],
schedule=Cron["0 6 * * *"],
] Incomplete cycles
A forecast can only be scored once its valid day's analysis is fetched, so a cycle's
7-day forecast waits a week. Hindcast handles that without dependencies between
partitions. A run's skill step works out the valid days its cycles
touch: each cycle's own day, whose analysis scores the forecasts of the seven cycles
before, and the seven days after, where its own forecasts are scored if those
analyses are already there. For each such day it recomputes everything valid that
day from hindcast.t2m and replaces the day's partition. A forecast whose
analysis isn't there yet is skipped, and scored by the run that fetches it.
So the scores don't depend on the order cycles arrive in: daily runs, a backfill in
any order, or a rerun all end with every forecast scored whose two cycles are
fetched. A cycle is complete once all seven leads are scored, and
hindcast report lists the cycles that aren't yet.
A cycle that isn't in the bucket yet fails the worker with "not published yet". For a
cycle of the last two days the step then defers half an hour with
morca.defer, spending no retry; for an older one it fails.
Running it
From the root of the morca repository, in the
rust pixi environment:
pixi shell -e rust
export CARGO_TARGET_DIR=$PWD/build/cargo
cargo build --release --manifest-path examples/hindcast/Cargo.toml
mkdir -p build/hindcast
scripts/embed-json.sh examples/hindcast/schemas/t2m.json T2M_SCHEMA > build/hindcast/t2m_schema.mojo
scripts/mojo-build.sh -I src -I build/hindcast -I examples/hindcast $DEP_INC \
examples/hindcast/hindcast.mojo -o build/hindcast/hindcast
export MORCA_WAREHOUSE=$PWD/build/hindcast/wh
export HINDCAST_LIB=$PWD/build/cargo/release/libhindcast_worker.so # .dylib on macOS
./build/hindcast/hindcast run --partitions 2026-11-01..2026-11-12
./build/hindcast/hindcast report 2026-11-01
That run of twelve cycles fetches 96 messages, about 7 MB, and scores what they
allow: the first five cycles completely, the later ones as far as 2026-11-12's
analysis goes. hindcast report [cycle] prints the skill per lead, the
cycles still waiting for analyses, the tables' row counts, and with a cycle, its
rows and city scorecard. HINDCAST_SOURCE points the worker at another
base URL or a directory laid out as the bucket, and HINDCAST_LEADS picks
the leads.
A 30-day backfill
Register the pipeline with a scheduler, which needs a database
(MORCA_DATABASE_URL), and backfill the days. The scheduler's runs inherit
its environment, so set MORCA_WAREHOUSE and HINDCAST_LIB
where it runs:
pixi run morca # builds ./build/morca
./build/morca register ./build/hindcast/hindcast
./build/morca backfill hindcast 2026-10-01..2026-10-30
./build/morca scheduler
./build/morca status hindcast
./build/hindcast/hindcast report
The backfill queues one run per day. Each fetches its cycle with eight Range requests
and rescores the valid days it touches, so the scores fill in as runs finish,
whatever order they finish in. Cycles to 2026-10-23 are complete when the backfill is
done; the last seven wait for November's analyses, which the daily schedule or
another backfill brings. morca backfill --rerun hindcast <days>
fetches and scores days again, replacing what was there.
Real numbers
A backfill of 13 cycles, 2026-10-31 to 2026-11-12, against the live bucket: all 13
runs succeeded in about 4 s each, with a peak of 44 to 63 MB. The tables ended with
6,776,640 rows in t2m, 63 in skill and 630 in
cities; 6 cycles were complete and the last 7 waiting for analyses, as
designed (#423). The skill per lead is the
mean bias and MAE over the cycles scored, and the RMSE over all of them:
| Lead | Cycles | Bias (K) | MAE (K) | RMSE (K) |
|---|---|---|---|---|
| 24h | 12 | −0.0147 | 0.3685 | 0.6464 |
| 48h | 11 | −0.0297 | 0.5403 | 0.9343 |
| 72h | 10 | −0.0397 | 0.6743 | 1.1684 |
| 96h | 9 | −0.0412 | 0.8028 | 1.3879 |
| 120h | 8 | −0.0563 | 0.9450 | 1.6500 |
| 144h | 7 | −0.0553 | 1.0928 | 1.8927 |
| 168h | 6 | −0.0666 | 1.2441 | 2.1848 |
The error grows steadily with lead, from 0.65 K a day ahead to 2.18 K a week ahead, and the bias stays small and slightly cold. Single cities vary far more: the 2026-11-01 forecast for 168 hours ahead was off by −6.7 K in São Paulo and +2.3 K in New York.
Tests
pixi run -e rust test-hindcast runs offline, in CI, on four real 74 KB
messages cut from NOAA's files. It checks the .idx lookup and the decode
against ecCodes, the library's exports and that it links no Rust std or OpenSSL,
that two cycles run in either order give the same scores, the skill and city numbers
against ecCodes and numpy, and that an incomplete cycle is reported, a rerun
replaces, and a missing cycle defers or fails by its age. The
README and the
code have the rest.