{% extends "backcast/base.html" %} {% load humanize %} {% block title %}Method ยท Backcast{% endblock %} {% block content %}

Method

Backcast asks one question of every stock: did any model forecast its price better than assuming nothing changes?

The benchmark

The naive forecast says the close in h trading days will equal today's close. For prices that behave like a random walk it is the best forecast available, which is why it is the bar every model has to clear. Beating it consistently is rare.

Rolling origins

Backcast loads five years of daily closes and makes a forecast from each of the last {{ origins }} trading days, about two years. On each of those days, every model sees all the prices up to and including that close, with at least {{ min_history }} days of history, and forecasts the close h trading days later.

Models are re-estimated every {{ refit }} trading days. In between they keep their estimated parameters but still see each new close, the way a model deployed in practice is used. Re-estimating every day was tested on AAPL, KO and NVDA at all three horizons, with closes to 23 October 2026: it moved each model's error relative to naive by 0.2 percentage points on average and 1.3 at most, changed none of the verdicts, and took 17 times as long. Re-estimating weekly came no closer to it. The check is scripts/refit_check.py in the repository, so anyone can run it again.

An earlier version used 30 non-overlapping windows. Shifting those windows by one day was enough to turn a significant result for one stock into noise, so the sample is now every trading day.

The closes are adjusted for stock splits but not for dividends. Dividend adjustment rescales every earlier price each time a dividend is paid, so the history would no longer show the prices people actually traded at. While a market is open, today's price is left out: it isn't a close until the session ends.

Scoring

Every metric is computed at the horizon endpoint only, on prices the model had not seen. Errors are measured on log prices: a miss of 2% counts the same whether the stock trades at 20 or at 2,000, so a stock that doubled over the two years doesn't weigh its later days twice as heavily.

The models work on log prices too, and their forecasts are converted back to prices. A price can't fall below zero, and it tends to move in proportion to its level, so a model fitted to raw prices can forecast impossible prices for a stock that has fallen a long way: on raw prices, the intervals for a stock that lost 99% of its value reached below zero on every day. Converting back keeps every forecast positive, and it makes each model's point forecast the median of its distribution, which is the value that minimises absolute error.

Mean error
The mean absolute difference between the log of the forecast and the log of the actual close. For errors of a few percent it is very close to the percentage error.
vs naive
The model's mean error divided by naive's, shown as a percentage change. The leaderboard is sorted by it.
CRPS vs naive
The continuous ranked probability score judges the whole forecast distribution. Every model gives nine quantiles, from 10% to 90%, and the score is twice their average pinball loss, which approximates CRPS. For a forecast with no spread at all it equals the absolute error, so it shares the mean error's units. Quantiles beyond 10% and 90% aren't scored.
Interval score vs naive
Gneiting and Raftery's score for the {{ level }}% interval: its width, plus ten times the distance by which the actual close falls outside it. A narrow interval only pays if it also holds the actual close.
Days won
The share of forecasts in which the model's error was smaller than naive's. On days a model forecast exactly what naive did, which ARIMA often does when it settles on a random walk, it neither won nor lost; the share of those ties is shown below. A model that tied on every day gets no p-value.
Direction right
How often the forecast moved in the same direction as the price. Forecasts of no change at all are left out. Most stocks rose more often than they fell over these years, so the results also give the score of always predicting a rise.
In {{ level }}% interval
How often the actual close fell inside the model's {{ level }}% prediction interval. Well calibrated intervals land near {{ level }}%.

Testing the margin

Forecasts made on neighbouring days overlap: two 20-day forecasts made a day apart share 19 days of the future, so their errors move together. A plain t-test treats them as independent and finds edges that aren't there. In a simulation with no real edge, it called 20-day results significant two times out of three.

For each model, Backcast runs the Diebold-Mariano test on the daily differences between naive's error and the model's. Its variance allows for correlation over h − 1 days, with Harvey, Leybourne and Newbold's small-sample correction. In the same simulation it was wrong 5 to 7% of the time at the 5% level, close to what it should be. {{ rivals|apnumber|capfirst }} models are compared with naive, so these p-values are also adjusted with Holm's method.

The headline asks a different question: did any of the {{ rivals|apnumber }} models beat naive? Picking the best of them and testing it alone overstates the evidence, because the best is the one most likely to have been lucky. Hansen's test for superior predictive ability (SPA) answers the question directly. It bootstraps the daily error differences of all the models together, resampling runs of neighbouring days so the overlap is kept, and asks how often the best model would look this good if none of them were better than naive. The headline calls a result significant only when that p-value is below {{ significance }}.

Trends

A significant result only means the model did better over this particular period. Some models forecast the same direction almost every time: Drift, for one, always predicts the stock's long-run average move. While a price keeps rising, a model like that beats naive day after day without forecasting anything. In a falling market it would lose just as steadily.

So the results split every model's error by the market it forecast in: days when the close was above its {{ trend_days }}-day average, and days when it was below. That is known on the day of the forecast, so the split shows whether an edge held in both kinds of market, not whether a model happened to guess the direction that came next. And when the best model predicted the same direction on at least {{ one_way }} of days and the price moved that way over the period, the headline says so.

Volatility

The direction of a price is close to a coin toss. The size of its moves isn't: big moves come in clusters, so a volatile week tends to follow a volatile week, and over months volatility drifts back to its usual level. That makes volatility the part of a stock's future that models can forecast, so Backcast has a second backtest for it, on the same {{ origins }} trading days, with the models re-estimated every {{ refit }} days.

Each model forecasts the variance of the return over the next h trading days. The forecast is scored against the realized variance, the sum of the squared daily log returns over those days. For one day that is a single squared return, a very noisy measure of how volatile the day really was. The score is QLIKE, log f + realized / f. Patton (2011) showed that with a noisy realized measure, QLIKE and squared error on the variance still rank forecasts the way the true variance would, while errors on volatility itself, the square root, do not. Of those two, QLIKE depends only on the ratio of realized to forecast, so a few violent days don't decide the result, and it charges more for forecasting too little volatility than too much.

Naive is the volatility of the last 20 trading days, the historical volatility traders quote. A month is a small sample, though, and volatility drifts back to its average, so a second baseline, the average over all the history before each forecast, is often harder to beat. The headline runs Hansen's SPA test against each.

Beating naive is not enough on its own. A model can win simply by averaging more days than naive's month, which needs no clustering at all. So scripts/volatility_check.py runs the volatility backtest on every stock in the S&P 500 study, and again on each stock's daily returns in shuffled order, which keeps every move but breaks up the calm and volatile stretches.

Across 498 stocks, some model beat naive on 38% of them at one day, 45% at five days and 56% at twenty. On the shuffled returns it did so far more often, on 75 to 84%: with no clusters to follow, a month of returns is just a noisier average than the whole history. So naive's month does pick up real clustering, and beating it proves little.

Against the long-run average, some model won on 4.4%, 3.2% and 4.4% of stocks at the three horizons, against 0%, 0.4% and 2.2% on the shuffled returns. That is more than the control, but no more than a test at the 5% level flags by chance. The long-run average was itself the best forecast on 105 stocks at one day and 144 at twenty; otherwise HAR and GJR-GARCH came out best most often. Volatility clustering is real, but over two years of daily closes the models that follow it rarely beat the plain average by more than luck on any one stock. Studies that find a clear edge measure each day's volatility from intraday prices, which is far less noisy than one squared daily return.

Limits

{% endblock %}