Validation dossier: Demo momentum (synthetic data)
Dossier 6a6bc62312579a29 · certificate 2db49c5a3e7b29c2 · 2026-10-12T14:46:03+00:00
Verdict: REJECT
Purpose and scope
| Name | Demo momentum (synthetic data) |
|---|---|
| Owner | Research |
| Purpose | Show the dossier format |
| Scope | One synthetic instrument, minute bars |
| Tier | 3 |
Summary
Why:
- net_profitability: net total return -21.27% after costs
- deflated_sharpe: deflated Sharpe 0.000 over 12 trial(s)
What to do:
- The strategy loses money after costs: reduce turnover or find a stronger edge.
- Sharpe does not survive the number of variants tried: test out-of-sample or reduce the search space.
Alpha Readiness Level
ARL 1/9 — not assessed beyond 1: the backtest is not broken (lookahead_truncation (not run), universe_lookahead (not run))
Rates the evidence, not the investment. Next: monte-neo verify --strategy (probes need code), or alpha with --membership
| ARL | Level | State | Blocking |
|---|---|---|---|
| 1 | Idea on clean data | pass met | |
| 2 | The backtest is not broken | — not assessed | lookahead_truncation (not run), universe_lookahead (not run) |
| 3 | It earns after costs | fail failed | net_profitability |
| 4 | The signal predicts | — not assessed | ic (not run), signal_placebo (not run), timing_significance (not run) |
| 5 | It is new alpha | — not assessed | hidden_short_volatility (not run), residual_alpha (not run) |
| 6 | It survives the search | fail failed | deflated_sharpe |
| 7 | Tradable and robust | — not assessed | delay_sensitivity (not run), period_consistency (not run), start_date_sensitivity (not run), cost_margin (not run), rebalance_timing_luck (not run) |
| 8 | It adds to your book | — not assessed | marginal_sharpe (not run) |
| 9 | Confirmed live | — not assessed | alpha_death_monitor (not run), live_vs_backtest (not run) |
Results
total_return-0.2127
sharpe_annualized-151.3
deflated_sharpe0
max_drawdown0.2161
breakeven_cost_bps0
n_closed_trades116
Checks
| check | category | status | summary |
|---|---|---|---|
| data_integrity | integrity | pass | OHLCV is clean |
| data_quality | integrity | pass | no spikes, frozen prices, split jumps or gaps |
| determinism | integrity | skip | determinism: needs strategy code (signal function) |
| lookahead_truncation | lookahead | skip | truncation probe: needs strategy code (signal function) |
| lookahead_perturbation | lookahead | skip | future-perturbation probe: needs strategy code (signal function) |
| external_data | lookahead | skip | outside data: needs strategy code (signal function) |
| lookahead_static_lint | lookahead | skip | static lint: no source provided |
| implausible_accuracy | lookahead | pass | next-bar hit rate 0.501 over 1179 bars (z 0.1) |
| data_independence | lookahead | skip | needs strategy code (positions only) |
| implausible_performance | statistics | pass | annualized Sharpe -151.3 is within the usual range |
| costs_modeled | economics | pass | 10.00 bps per side modeled |
| net_profitability | economics | fail | net total return -21.27% after costs |
| cost_margin | economics | skip | not profitable after costs |
| delay_sensitivity | economics | skip | not profitable after costs |
| timing_significance | statistics | skip | not profitable after costs |
| sample_size | statistics | pass | 116 closed trades (min 30) |
| deflated_sharpe | statistics | fail | deflated Sharpe 0.000 over 12 trial(s) |
| trials_disclosed | statistics | pass | n_trials declared |
| holdout_consistency | statistics | pass | Sharpe/bar train -0.2024 vs holdout -0.2256 |
| period_consistency | statistics | skip | not profitable after costs |
| walk_forward_stability | statistics | info | 0 of 6 equal windows (190 bars each) made money; worst -6.1%, best -1.1% |
| sharpe_confidence | statistics | info | Sharpe 95% interval -205.82 to -98.37 (return -28.4% to -13.8%); 0% of resamples have a positive Sharpe |
| serial_correlation | statistics | info | serial correlation of returns is small (lag-1 +0.23): Sharpe -151.33, corrected -111.86 |
| track_record | statistics | info | the Sharpe is not positive: no history length makes it significant |
| benchmark | statistics | info | buy & hold +0.59% (Sharpe 4.88) vs strategy -21.27% (Sharpe -151.33) |
| spread_estimate | economics | info | rough spread estimate from high and low: about 9.82 bps (4.91 bps per side); the model charges 5 bps slippage per side |
| capacity | economics | info | capital that keeps 90% of 115 fills within a share of the bar's volume: 1%: 2.02K, 5%: 10.1K, 10%: 20.2K (median bar trades 2.55M) |
Framework map
Federal Reserve SR 11-7 / OCC 2011-12, Supervisory Guidance on Model Risk Management. Coverage means evidence in this dossier, not compliance: clauses marked “needs a person” require a judgement the verifier cannot make.
| Clause | Coverage | Result | Evidence |
|---|---|---|---|
| Conceptual soundness Design, theory and logic of the model are sound | partial needs a person | pass | data_integrity, data_quality, implausible_accuracy |
| Data quality and relevance Data are suitable and their limitations are understood | covered | pass | data_integrity, data_quality |
| Outcomes analysis and back-testing Model outputs are compared with outcomes | covered | fail | benchmark, capacity, costs_modeled, deflated_sharpe, holdout_consistency, implausible_performance, net_profitability, sample_size, serial_correlation, sharpe_confidence, spread_estimate, track_record, trials_disclosed, walk_forward_stability |
| Sensitivity analysis Results are tested under changed inputs and assumptions | not covered | — | |
| Assumptions and limitations They are identified and documented | covered | pass | limitations |
| Effective challenge Independent, competent review that can be reproduced | partial needs a person | pass | repro |
| Ongoing monitoring Performance is monitored after implementation | not covered | — | |
| Governance, policies and controls Limits and policies are defined and applied | not covered needs a person | — | |
| Documentation Enough detail for a third party to understand the model | covered | pass | limitations, repro |
Assumptions and limitations
- Fills at the next bar's open; commission 5.0 bps and slippage 5.0 bps per side.
- Not checked: determinism (determinism: needs strategy code (signal function)).
- Not checked: lookahead_truncation (truncation probe: needs strategy code (signal function)).
- Not checked: lookahead_perturbation (future-perturbation probe: needs strategy code (signal function)).
- Not checked: external_data (outside data: needs strategy code (signal function)).
- Not checked: lookahead_static_lint (static lint: no source provided).
- Not checked: data_independence (needs strategy code (positions only)).
- Not checked: cost_margin (not profitable after costs).
- Not checked: delay_sensitivity (not profitable after costs).
- Not checked: timing_significance (not profitable after costs).
- Not checked: period_consistency (not profitable after costs).
- The verifier tests the backtest's methodology on the data given; it does not forecast future returns.
- The economic rationale of the idea (why it should work) is a human judgement; the dossier gives evidence, not that judgement.
- Costs, fills and latency are models: queue position and partial fills are not modelled.
- A strategy that behaves differently when it detects a test can only be caught by the probes it does not detect.
Reproduce
monte-neo verify --recheck certificate.json --ohlcv DATA.csv --strategy STRATEGY.py
Python 3.13.16 · macOS-27.0.1-arm64-arm-64bit-Mach-O · numpy 2.3.5, pandas 3.0.0, numba 0.63.1, monte-neo 0.53.0
Sign-off
| Role | Name | Date | Signature |
|---|---|---|---|
| Model developer | |||
| Independent validator | |||
| Model owner |
Not investment advice. Monte-Neo is a research and verification tool. Its reports, examples and reference strategies illustrate a method; they are not a recommendation, an offer or a call to buy, sell or trade any instrument, and not a promise of profit. Simulated and past results do not predict future returns. Trading can lose money, including more than the initial capital when leverage is used. Decisions and their consequences are yours; consult a licensed adviser if you need advice.