A curated, hash-pinned suite of (strategy, dataset, expected report) cases. Run one and it either reproduces the frozen report byte-for-byte, in ten languages, or the build goes red.
Every curated case, recomputed and checked against its frozen report and hash:
wickra-benchmark run-suite --suite cases/suite.json --data-root datasetsid passed hash_match hash
breakout-channel-01 true true 2b1ef11f989c
buy-and-hold-01 true true c1f6820a3de2
ema-trend-follow-01 true true 97a97c31a400
rsi-mean-reversion-01 true true 664558550a58
sma-crossover-01 true true 8f5e84ff8862
5/5 passedExit code 0 means every case reproduced, 1 that at least one did not — so a drifting engine turns a build red rather than going unnoticed.
pip install wickra-benchmarkFour things frozen together, and a fifth recomputed from them:
| Field | What it holds |
|---|---|
strategy | An embedded wickra-backtest StrategySpec, as raw JSON. |
dataset_ref | The deterministic candle CSV the case runs on. |
expected | The BacktestReport the engine must produce, byte for byte. |
expected_hash | The lowercase blake3 of that report's canonical form. |
Running the case recomputes the report from strategy plus the dataset and returns two independent booleans: passed (the recomputed report equals expected) and hash_match (its canonical hash equals expected_hash).
They are separate on purpose. Both false is an engine that computes something different. Only hash_match false means the case's own two expectations disagree — which happens when a case is hand-edited instead of blessed.
Because nothing is reimplemented. Every binding calls the same Rust core and returns its canonical string verbatim — sorted keys, round-trippable floats — so the response from R and the response from Rust are the same bytes, not merely the same numbers to some tolerance.
That is not free, and the corpus keeps it deliberate: a case may name any indicator the engine offers, and some of those call a transcendental from the platform's math library. No mainstream libm rounds those correctly, and implementations differ in the last bit. A case built on one would have to compare to a relative tolerance instead. None currently does.
Pre-release. The machinery is complete and the corpus is not: five cases over five deterministic datasets of 60–80 bars, which is enough to prove that a recompute reproduces and not yet enough to call a benchmark suite. Growing it — more regimes, longer series, more strategy families — is the work before 1.0.