Benchmarks & synthetic data¶
Under testing
These benchmarks are how ClimAID is currently tested. Because the data are synthetic, good results here are necessary but not sufficient: real-data validation is still to come.
All synthetic datasets are generated by benchmarks/generators.py in the repository, with fixed random seeds,
so every result can be reproduced exactly. Each dataset has a known truth, so model output can be checked
against the right answer.
Datasets¶
| Generator | What it contains | Used to test |
|---|---|---|
climate() |
Monthly climate 2004–2022: seasonal temperature, monsoon-shaped rainfall, humidity, slowly varying ENSO | Shared input |
seasonal() |
Cases driven by temperature (1 month earlier), rainfall (2 months) and ENSO (3 months); calendar month explains ~70% of variation | Forecasts, lag recovery |
nonseasonal() |
Cases driven only by ENSO (4 months earlier) plus slow drift; no annual cycle | Forecasts when seasonality is absent |
enso_interaction() |
Seasonal truth plus El Niño amplifying the temperature effect | ENSO interaction features |
realistic_hard() |
Seasonal climate response with a threshold rainfall effect, saturating temperature effect, a reporting change (+60% from 2016), two serotype-driven outbreak years, a 2020 reporting collapse, ~5% missing months plus a 3-month gap, and two data-entry errors (10×) | Robustness to real-world data problems |
cmip6() |
Three imaginary climate models (own biases and warming sensitivities) × SSP2-4.5 / SSP5-8.5, 2015–2060 | Scenario outlook, bias correction |
multi_district() |
Six districts sharing one climate response but with different seasonal climates | Multi-district pooling |
thermal_optimum() |
A hot district whose transmission follows a thermal-suitability curve | Temperature-curve option |
Counts are drawn with more scatter than a simple Poisson model (negative binomial), as in real surveillance data.
Why clean synthetic data flatters the models¶
The clean datasets are built the way the models assume the world works: smooth climate effects, fixed lags
inside the search ranges, a relationship that never changes, and clean noise. Real data also have reporting
changes, testing campaigns, new serotypes, gaps and errors. realistic_hard() adds these, and is the more
informative test.
Running the benchmark¶
python benchmarks/run_benchmarks.py --tuning fast
It makes 12-month forecasts from four start dates for each dataset through the full forecast_v2() path
(tuning, backtests, calibration) and writes benchmarks/results/summary.csv and summary.md. The key column,
WIS_vs_seasonal_naive, compares each model's probabilistic score with the "same as recent years" baseline:
below 1.0 means better than the baseline.
Results are summarised on the Status & validation page.
--models a,b,... chooses the models (default: seasonal naive, Poisson, random forest, gradient boosting).
SARIMAX¶
SARIMAX was benchmarked alongside the four standard benchmark models (same run, Fast tuning, 12-month forecasts
from four start dates; results in benchmarks/results/sarimax/):
python benchmarks/run_benchmarks.py --tuning fast \
--models seasonal_naive,poisson,random_forest,gradient_boosting,sarimax --out benchmarks/results/sarimax
Score relative to the baseline (WIS ratio; lower is better) and coverage of the 80% likely range:
| Dataset | SARIMAX | Best other single model | Ensemble (with SARIMAX) | SARIMAX 80% coverage |
|---|---|---|---|---|
| Seasonal (clean) | 0.63 | 0.63 (random forest) | 0.60 | 0.98 |
| Non-seasonal (clean) | 0.58 | 1.04 (Poisson) | 0.86 | 0.94 |
| Realistic (hard) | 0.84 | 0.63 (Poisson) | 0.63 | 0.72 |
SARIMAX was the best single model on the non-seasonal data, where the other models were no better than the baseline, and tied for best on the seasonal data. On the realistic dataset (reporting change, outbreak years, missing months, data-entry errors) it was clearly worse than Poisson regression and the tree models, and its 80% ranges contained the truth only 72% of the time. Adding it improved the ensemble slightly on the clean datasets and left it unchanged on the realistic one. Because the realistic case is the closest to real surveillance data, SARIMAX is offered as an optional model and benchmark rather than a default; this can be revisited once ClimAID has been tested on real data.