Skip to content

Benchmarks & synthetic data

Under testing

These benchmarks are how ClimAID is currently tested. Because the data are synthetic, good results here are necessary but not sufficient: real-data validation is still to come.

All synthetic datasets are generated by benchmarks/generators.py in the repository, with fixed random seeds, so every result can be reproduced exactly. Each dataset has a known truth, so model output can be checked against the right answer.

Datasets

Generator What it contains Used to test
climate() Monthly climate 2004–2022: seasonal temperature, monsoon-shaped rainfall, humidity, slowly varying ENSO Shared input
seasonal() Cases driven by temperature (1 month earlier), rainfall (2 months) and ENSO (3 months); calendar month explains ~70% of variation Forecasts, lag recovery
nonseasonal() Cases driven only by ENSO (4 months earlier) plus slow drift; no annual cycle Forecasts when seasonality is absent
enso_interaction() Seasonal truth plus El Niño amplifying the temperature effect ENSO interaction features
realistic_hard() Seasonal climate response with a threshold rainfall effect, saturating temperature effect, a reporting change (+60% from 2016), two serotype-driven outbreak years, a 2020 reporting collapse, ~5% missing months plus a 3-month gap, and two data-entry errors (10×) Robustness to real-world data problems
cmip6() Three imaginary climate models (own biases and warming sensitivities) × SSP2-4.5 / SSP5-8.5, 2015–2060 Scenario outlook, bias correction
multi_district() Six districts sharing one climate response but with different seasonal climates Multi-district pooling
thermal_optimum() A hot district whose transmission follows a thermal-suitability curve Temperature-curve option

Counts are drawn with more scatter than a simple Poisson model (negative binomial), as in real surveillance data.

Why clean synthetic data flatters the models

The clean datasets are built the way the models assume the world works: smooth climate effects, fixed lags inside the search ranges, a relationship that never changes, and clean noise. Real data also have reporting changes, testing campaigns, new serotypes, gaps and errors. realistic_hard() adds these, and is the more informative test.

Running the benchmark

python benchmarks/run_benchmarks.py --tuning fast

It makes 12-month forecasts from four start dates for each dataset through the full forecast_v2() path (tuning, backtests, calibration) and writes benchmarks/results/summary.csv and summary.md. The key column, WIS_vs_seasonal_naive, compares each model's probabilistic score with the "same as recent years" baseline: below 1.0 means better than the baseline.

Results are summarised on the Status & validation page.