Metadata-Version: 2.5
Name: forelint
Version: 0.1.0
Summary: Lint your forecast backtests: leakage between related series, the run-to-run noise floor, model comparison tests, and per-item bias before you trust a score.
Project-URL: Homepage, https://github.com/davidmashiah/forelint
Project-URL: Source, https://github.com/davidmashiah/forelint
Project-URL: Author, https://davidmashiah.com/
Author: David Mashiah
License-Expression: MIT
License-File: LICENSE
Keywords: backtesting,data leakage,demand forecasting,diebold-mariano,forecast bias,forecast evaluation,forecast value added,forecasting,group k-fold,model confidence set,time series cross-validation,tracking signal
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.9
Requires-Dist: numpy>=1.22
Requires-Dist: pandas>=1.5
Requires-Dist: scikit-learn>=1.2
Requires-Dist: scipy>=1.9
Provides-Extra: test
Requires-Dist: arch>=6; extra == 'test'
Requires-Dist: pytest>=7; extra == 'test'
Description-Content-Type: text/markdown

# forelint

**Lint your forecast backtests before you trust a score.**

forelint checks the things a backtest score hides: leakage between related
series, the run-to-run noise floor, whether model A is really better than
model B, and which items carry the bias and the error. It works on the
long-format tables that statsforecast, mlforecast and most forecasting code
already produce, and on any scikit-learn estimator.

```bash
pip install forelint
```

## What it checks

| Question | Function |
|---|---|
| Does my validation leak through related stores, products or regions? | `split_optimism`, `GroupTimeSplit` |
| Is this improvement bigger than seed-to-seed noise? | `noise_floor` |
| Is model A really better than B across many series? | `dm_test`, `diebold_mariano`, `model_confidence_set` |
| Which items are always over or under forecast? Does the model beat a naive forecast? Where does the error live? | `audit` |

## Quickstart

```python
import forelint as fl

# df: one row per item and date, the actual in "y", one column per model,
# e.g. the output of statsforecast's cross_validation()
report = fl.audit(df, models=["LightGBM", "ETS"], baseline="SeasonalNaive")
print(report)

# Is LightGBM really better than the seasonal naive, or is it noise?
print(fl.dm_test(df, "LightGBM", "SeasonalNaive"))
```

```python
# How much does random K-fold flatter the model, compared with holding out
# the future of entities the model never saw?
fl.split_optimism(model, X, y, groups=store_ids, time=dates)

# What difference between two runs is just noise?
nf = fl.noise_floor(lambda seed: train_and_score(seed), seeds=range(10))
nf.is_meaningful(new_mae - old_mae)
```

## On real data: one Walmart store in M5

`benchmarks/m5_benchmark.py` runs every check on the public M5 data: store
CA_1, department FOODS_3, 823 items, the last 365 days, a direct 28-day-ahead
gradient-boosting model from scikit-learn, scored on the final 28 days.
Measured, not assumed:

- **The model added nothing over a 28-day moving average.** Forecast value
  added was **−0.1%** on MAE, the model was worse than the moving average on
  **49% of items**, and Diebold-Mariano gives **p = 0.31**. The model
  confidence set keeps both. An average-MAE table alone would not show this.
- **It under-forecasts by 9.3%** of actual sales.
- **10% of items hold 73% of the squared error.**
- **The noise floor is small here:** over 10 seeds the MAE sd is 0.0014, so
  any change under 0.0045 is noise.
- **Splitting barely mattered for this data:** random K-fold was 1% optimistic
  against a time split. Leakage depends on the features; forelint measures it
  rather than assuming it.

The model is deliberately plain (six features, no tuning). The point is the
audit, not the model.

## Why

In an international machine-learning competition on well-geology prediction,
validation ranked five builds against the hidden leaderboard with a
correlation of −0.745: the build it liked most scored worst, and identical code scored 6.241, 6.259 and
6.368 on three runs. Both failures were measurable before the deadline with
the right checks. forelint is those checks, packaged.

Background: [My validation ranked five models backwards](https://davidmashiah.com/notes/validation-ranked-my-models-backwards).

## Conventions

- Error is **forecast − actual**. Positive bias means over-forecasting.
- Tracking signal is the sum of errors divided by the mean absolute error;
  the default alarm level is ±4.
- `dm_test` averages the loss difference across series at each time stamp,
  then applies Diebold-Mariano with the Harvey-Leybourne-Newbold correction.
- `model_confidence_set` follows Hansen, Lunde and Nason (2011) with the
  T_max statistic and a moving-block bootstrap. Its kept set matches
  `arch.bootstrap.MCS` in the test suite.

## Status

Alpha. The API may change before 1.0. Issues and pull requests are welcome.

## Author

[David Mashiah](https://davidmashiah.com). MIT licensed.
