How to verify a point-in-time backtest from data to portfolio

A completed backtest proves that a process returned a result. Establishing that it traded the intended historical universe and applied the intended economics requires independent checks of the data, state transitions, and persisted portfolio.

Stratifyre’s recent universe acceptance work followed actual production Rust jobs through the deployed API, workflow, data-loading, execution, and replay path. The acceptance record contains 24 positive jobs, eight frozen reruns after a genuine reference-head advance, and retained original failed-oracle receipts. The failures helped identify what a successful completion flag could not establish.

Define the financial answer before running the engine

An independent oracle is an expected result calculated from declared inputs and terms. It should not obtain its answer by calling the same implementation it is checking. Otherwise, a shared mistake can make the result and its test agree.

For a split, declare the initial shares, basis, cash, and pending orders. For a merger, declare consideration, successor identity, timing, and the simulator’s basis and fractional-unit policies. Freeze those inputs before looking at the output.

Seeded holdings make these controls meaningful. A merger test with no shares at the event boundary can “pass” without applying a settlement. A rename test with no pending order or active evaluator state does not establish that those owners survived. Each case needs an initial state that exercises the claimed behavior.

Our AAPL control expected 10,000 shares to become 40,000 while conserving $500,000 lot basis and $1,000,000 cash. The XLNX stock control expected 17,234 AMD shares valued at $112.30. Independent arithmetic gives market value of $1,935,378.20; with the control’s $1,000,000 cash, equity is $2,935,378.20. Production GSPC and explicit controls passed that original oracle after the runtime repair.

These are financial consistency tests under declared models. They do not forecast strategy returns or establish that every source event has sufficient settlement terms.

Test different contracts with different controls

An explicit-instrument control isolates financial execution. A historical-universe control adds membership and identity resolution. A static basket tests an intentionally fixed selection. Combining a universe with an overlapping basket exercises ownership and duplicate handling.

This matrix prevents a narrow success from expanding into a broader claim. Our AET mixed-merger positive case used an explicit instrument because the wider GSPC interval retained membership and identity gaps. The pending AAPL split order used a qualified static basket because the wider universe interval contained an unqualified dividend share frame.

Negative controls are part of acceptance. The historical NDX request remained unavailable and its admission rejection was retained. A qualified system must be able to reject an experiment whose data contract it cannot fulfill.

Preserve failures and repair the responsible layer

The original GSPC stock-merger result failed its financial oracle. The repair addressed stale successor valuation outside regular trading hours, reuse of delivered ownership through a rename, and handling of successor quotes before a holding existed. It preserved strict errors for missing owners and the runtime’s progress and cancellation callbacks.

That failure demonstrated why checking delivered shares alone was insufficient. Correct share quantity combined with a stale price still produces incorrect equity. The repaired stock rerun retained the original configuration, seed, strategy version, price hashes, and oracle.

A separate All/Once targeting fixture also needed correction. The targeted rename control then retained its single one-share fill at $195.10. A fixture error and a runtime error require different repairs; changing expected economics to match an incorrect engine result would conceal the latter.

The original failed-oracle receipts remained available. Across the test jobs, all 26 owning workflows reached completion, including 24 positive acceptance jobs and two retained original failed-oracle cases. Workflow completion and financial acceptance are distinct observations.

Advance the real head and rerun the frozen experiment

An immutable reference pin should keep an earlier run tied to its earlier history even after a newer provider revision becomes selected. A useful test needs an actual head change and a run that can accidentally read the new history.

Production acceptance published changed vendor ticker, action, and S&P 500 archives, advancing the reference head from v9 to v11 through the real publication workflow. The earlier revision’s rows, counts, and fingerprint remained unchanged, with its immutability protections enabled.

Eight frozen reruns then completed and passed their original economics. They retained the old pin, configuration, seeds, strategy version, and prices. Existing persisted replay remained unchanged, and canonical replay states from the reruns matched. An unpinned GSPC split request separately selected the actual new v11 head and passed its control.

Those observations test both sides of revision selection: a frozen run continues to read its selected history, while an unpinned run follows the selected head.

Reference revisioning has a precise limit. It does not version raw OHLC rows. Price hashes established unchanged inputs for these comparisons; future price corrections are not automatically frozen by a reference pin. A reproducibility record must therefore include price identity, not merely the reference revision ID.

Nor does this test create a historical knowledge archive. The source is reconstructed dated reference history. Reruns can reproduce that declared reconstruction while historical announcement-time knowledge remains a separate evidence requirement.

Cross-process proof exposes integration failures

The data adapter, database, API, workflow, cache, Rust engine, and replay store run across different process boundaries. They can disagree about timestamp units, serialized decimals, selected revisions, or which execution attempt owns an artifact.

A unit test can exercise an isolated transformation. A process test can exercise actual serialization and persistence. A production control can establish that the qualified deployed path connects those pieces. Each adds evidence that the earlier layer cannot supply by itself.

Retries make ownership especially consequential. A replaced worker must not publish an old attempt’s state into the current result. A settlement receipt must remain tied to the execution and revision that produced it. Reading an artifact without checking that ownership can return a plausible but stale portfolio.

Scale changes the engineering problem too. The accepted new reference revision contained 355,183 actions, 33,201 identity records, 1,414 membership records, and 247,873 coverage rows. Publication must preserve a consistent snapshot across those domains, while a short backtest should load the financial interval and dependencies it needs rather than materialize the entire corpus.

These counts describe the accepted snapshot. They are not throughput measurements, simultaneous universe sizes, or a promise of complete coverage.

Treat release proof and financial proof as separate gates

The acceptance record includes 1,052 Rust tests and 20 system suites comprising 322 tests, alongside runtime and NAPI performance checks. The accepted production source also passed required CI and the existing GitOps promotion process, with the deployed images verified against CI provenance.

That establishes the release path for the tested source. It does not replace financial controls on the deployed runtime. Healthy processes can still select the wrong history or value a successor incorrectly; local fixtures can pass while production reference data remains incomplete.

The supported result is bounded: qualified GSPC/static-basket cases and explicit financial controls passed under declared simulator models. Unsupported dates and event forms remain explicit. The source evaluation used an authorized personal key, and commercial entitlement remains a separate pending requirement.

Build an acceptance record you can inspect

For a consequential historical-universe result, retain:

1. The exact strategy, saved configuration, initial holdings, costs, and event model.
2. Dated membership and identity evidence, coverage decisions, reference pin, and price hashes.
3. Independent expected quantities, cash, basis, applicable quotes, and equity.
4. State around each event, including orders and evaluator history that cross the boundary.
5. Actual terminal outcomes and persisted artifacts from the owning execution attempt.
6. Frozen reruns after a genuine head advance, alongside a new-head control and rejected unsupported cases.

This is the engineering effort behind a trustworthy return curve: making the historical selection, runtime economics, and recorded explanation agree across the complete path. Start with the historical-universe data contract when deciding what your next backtest needs to preserve.