Benchmarks & synthetic data¶
Under testing
These benchmarks are how ClimAID is currently tested. Because the data are synthetic, good results here are necessary but not sufficient: real-data validation is still to come.
All synthetic datasets are generated by benchmarks/generators.py in the repository, with fixed random seeds,
so every result can be reproduced exactly. Each dataset has a known truth, so model output can be checked
against the right answer.
Datasets¶
| Generator | What it contains | Used to test |
|---|---|---|
climate() |
Monthly climate 2004–2022: seasonal temperature, monsoon-shaped rainfall, humidity, slowly varying ENSO | Shared input |
seasonal() |
Cases driven by temperature (1 month earlier), rainfall (2 months) and ENSO (3 months); calendar month explains ~70% of variation | Forecasts, lag recovery |
nonseasonal() |
Cases driven only by ENSO (4 months earlier) plus slow drift; no annual cycle | Forecasts when seasonality is absent |
enso_interaction() |
Seasonal truth plus El Niño amplifying the temperature effect | ENSO interaction features |
realistic_hard() |
Seasonal climate response with a threshold rainfall effect, saturating temperature effect, a reporting change (+60% from 2016), two serotype-driven outbreak years, a 2020 reporting collapse, ~5% missing months plus a 3-month gap, and two data-entry errors (10×) | Robustness to real-world data problems |
cmip6() |
Three imaginary climate models (own biases and warming sensitivities) × SSP2-4.5 / SSP5-8.5, 2015–2060 | Scenario outlook, bias correction |
multi_district() |
Six districts sharing one climate response but with different seasonal climates | Multi-district pooling |
thermal_optimum() |
A hot district whose transmission follows a thermal-suitability curve | Temperature-curve option |
Counts are drawn with more scatter than a simple Poisson model (negative binomial), as in real surveillance data.
Why clean synthetic data flatters the models¶
The clean datasets are built the way the models assume the world works: smooth climate effects, fixed lags
inside the search ranges, a relationship that never changes, and clean noise. Real data also have reporting
changes, testing campaigns, new serotypes, gaps and errors. realistic_hard() adds these, and is the more
informative test.
Running the benchmark¶
python benchmarks/run_benchmarks.py --tuning fast
It makes 12-month forecasts from four start dates for each dataset through the full forecast_v2() path
(tuning, backtests, calibration) and writes benchmarks/results/summary.csv and summary.md. The key column,
WIS_vs_seasonal_naive, compares each model's probabilistic score with the "same as recent years" baseline:
below 1.0 means better than the baseline.
Results are summarised on the Status & validation page.