{% extends "backcast/base.html" %} {% load humanize static %} {% block title %}Findings · Backcast{% endblock %} {% block description %}Every S&P 500 member backtested against the naive forecast, next to placebos that show what luck alone produces.{% endblock %} {% block head %} {% endblock %} {% block content %}

Findings

Every member of the S&P 500 was backtested against the naive forecast at three horizons. Each stock was also run on placebo copies of itself, where no model can have a real edge, to see how often the test finds a winner by luck alone.

{% if not summary %}

This copy of Backcast doesn't have the study's results yet. scripts/study.py produces them.

{% else %}{% with study=notes.study %}

What it found

None of the models beat naive by a useful margin. Out of 498 stocks, the SPA test flagged a winner on 8 at the one-day horizon, 2 at five days and 2 at twenty days. The placebos, where any win is luck, gave 1 to 5 winners at one day depending on the draw, 0 to 7 at five days and 2 to 20 at twenty days. All of this uses closes up to {{ study.end }}.

So at five and twenty days the real stocks did no better than the placebos. At one day, 8 is more than any placebo draw gave, but the wins are tiny. On those 8 stocks the best model's error was only 0.1% to 0.5% lower than naive's.

The test was stricter than its 0.05 cutoff suggests. On the placebos it flagged 0.4% to 1.7% of stocks on average, not the 5% you might expect, so the real results are compared with the placebos and not with 5%.

Seven of the eight one-day wins went to Theta or ETS, and they come from small reversals. Over these two years, a stock's daily move tended to be partly undone the next day. The median correlation between one day's return and the next was −0.024, and it was negative for 70% of stocks. Theta and ETS forecasts sit a little behind the latest price, so they gain when a move partly reverses. Theta's error was lower than naive's on 59% of the real stocks but only 33% of the placebos, where the random signs wipe out any reversal. At the median stock the difference was 0.02%, far too small to trade on.

Valero is a good example of a trend passing for skill. It won at one day and at twenty, both times with Drift, which assumes the recent trend continues, and its price roughly tripled over the period. At twenty days Drift's error was 11% lower than naive's when the price was above its {{ trend_days }}-day average, and 2% higher when it was below. The other twenty-day winner, Phillips 66, is also an oil refiner.

LightGBM, the only machine-learning model, came last. At the median stock its error was 2% to 3% higher than naive's, and the Diebold-Mariano test didn't put it ahead of naive on a single stock at any horizon. Drift should have been helped by survivorship bias, since every stock here made it into the 2026 index, but the same test put it ahead on just one stock.

How often a model won

Stocks where Hansen's SPA test found that the best of the {{ study_rivals|apnumber }} models beat naive, at p below {{ significance }}. The placebo figure is the average over the placebo draws, with the lowest and highest draw beneath it.

{% for row in summary %} {% endfor %}
Horizon Stocks Placebos Trend bets
{{ row.horizon }} trading day{{ row.horizon|pluralize }} {{ row.real.percent }}{{ row.real.significant }} of {{ row.real.tested }} {% if row.placebo.draws %}{{ row.placebo.percent }}{{ row.placebo.range }}{% else %}—{% endif %} {{ row.real.trend_bets }}of {{ row.real.significant }} wins

A trend bet is a win where the best model forecast the same direction from at least {{ one_way }} of the days, and the price went that way over the period. A model like that does well for as long as the trend lasts and badly once it reverses.

Where the p-values fell

The share of stocks in each band of SPA p-values, real stocks against placebos (all draws pooled). The first band, below {{ significance }}, holds the wins counted above. Bars that rise above the placebo marks there mean more evidence than chance produces.

{% for h in horizons %}

{{ h }} trading day{{ h|pluralize }}

{% endfor %}
Show the data

Each model on its own

Stocks where one model beat naive by the Diebold-Mariano test, with p below {{ significance }} after Holm's adjustment for {{ study_rivals|apnumber }} models. The smaller figure is the placebo share.

{% for h in horizons %}{% endfor %} {% for model in models %} {% for cell in model.cells %}{% endfor %} {% endfor %}
Model{{ h }} day{{ h|pluralize }}
{{ model.name }}{{ cell.real }}{{ cell.placebo }}

The stocks where a model won

Every real stock and horizon with a significant SPA result. The uptrend and downtrend columns split the best model's error by whether the close was above or below its {{ trend_days }}-day average on the forecast date. An edge that vanishes on one side is probably the trend. The links open today's backtest, which drifts from the study's as new closes come in.

{% if winners %}
{% for row in winners %} {% endfor %}
Stock Horizon Best model Error vs naive Uptrend Downtrend SPA p-value Price change
{{ row.symbol }}{{ row.name }} {{ row.horizon }} day{{ row.horizon|pluralize }} {{ row.best }}{% if row.trend_bet %}trend bet{% endif %} {{ row.change.text }} {{ row.up.text }} {{ row.down.text }} {{ row.spa_p_value }} {{ row.period_change }}
{% else %}

No stock had a significant result at any horizon.

{% endif %}

How the study was run

The universe is the {{ notes.universe.members }} members of the S&P 500 listed on Wikipedia on {{ notes.universe.retrieved }} (this revision). {{ study.tested }} of them were tested at one horizon or more; the first table counts each horizon. Every stock gets five years of daily closes ending {{ study.end }} and the same backtest as the rest of Backcast: a forecast from each of the last {{ study.origins }} trading days, models re-estimated every {{ study.refit }}, scored on prices they had not seen. The method page has the details.

A placebo replays a stock's own daily moves with random signs. Each move keeps its size, so the placebo is just as volatile as the real stock, with the same calm and wild stretches. Only the direction is random, and no model can forecast a coin toss. Each date gets one coin toss for all stocks, so stocks that moved together still do.

That also gives the whole market a single random path. If that path happens to trend, trend bets win on many placebos at once, so a single draw could make luck look more or less common than it is. The placebo was drawn {{ study.seeds|length|apnumber }} times, from the seeds {{ study.seeds|join:", " }}.

Limits

{% if study.excluded or study.not_loaded %}
Backtests that could not run
    {% for row in study.excluded %}
  • {{ row.symbol }} at {{ row.horizon }} day{{ row.horizon|pluralize }}{% if row.seed %}, placebo {{ row.seed }}{% endif %}: {{ row.reason }}
    • {% endfor %} {% for symbol in study.not_loaded %}
  • {{ symbol }}: the prices never loaded.
    • {% endfor %}
{% endif %}

Download the data

results.csv has a row for every stock, horizon, draw and model, with all the scores from the results page. study.json holds the settings and universe.csv the member list. Code version {{ study.code }}; scripts/study.py runs it again.

{{ chart|json_script:"findings-data" }} {% endwith %}{% endif %}
{% endblock %}