papers¶
Paper-replica shims — slot any new method onto each paper’s exact metric axis.
Paper-replica shims — evaluate any model under the exact metric set of a published paper, so cross-paper comparison becomes one-line.
Each module exposes one function metrics(…) returning the metrics the paper reports, in the paper’s preferred units. Internally each shim composes scitex_seizure_metrics primitives (detection.evaluate / forecasting.evaluate_stream / calibration.calibration_report / surrogates) — no re-implementation of the math.
Available shims: - cook2013 — first-in-man trial Snyder traffic-light (Lancet Neurology). - karoly2017 — circadian + iEEG forecasting (Brain). - kuhlmann2018 — Epilepsy Ecosystem clip-level AUC (Brain). - maturana2020 — critical-slowing biomarker (Nat Commun). - proix2021 — multi-day-horizon multidien forecasting (Lancet Neurology). - stirling2021 — sub-scalp 4-channel forecasting (Front Neurol). - andrade2024 — sample- vs alarm-based side-by-side (Front Neurosci).
Cook et al. 2013 (Lancet Neurology) replica metrics.
Paper: ‘Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study’. The clinical-trial seminal paper. Reports: - per-patient sensitivity at high-likelihood warning windows. - proportion of time in high / moderate / low warning (Snyder scheme). - no AUROC / AUPRC / FP/hr — clinical, not ML, framing.
Citation: Cook MJ et al., Lancet Neurol 2013; 12: 563–571. doi:10.1016/S1474-4422(13)70075-9
- scitex_seizure_metrics.papers.cook2013.metrics(*, y_proba, times_seconds, seizure_times, sph_seconds=0.0, sop_seconds=3600, high_threshold=0.8, moderate_threshold=0.5, n_surrogate=200, name='cook2013')[source]¶
Reproduce the Snyder-scheme time-in-warning panel.
- Parameters:
y_proba – per-window predicted probabilities.
times_seconds – matching timestamps.
seizure_times – seizure onset times.
sph_seconds (float) – prediction horizon.
sop_seconds (float) – occurrence period.
moderate_threshold (float) – lights cut-offs.
n_surrogate (int) – surrogates for IoC.
name (str) – identifier.
high_threshold (float)
moderate_threshold
- Returns:
dict with sensitivity_high (the warning that flagged the seizure), time_in_high_frac, time_in_moderate_frac, time_in_low_frac, ioc_high — mirroring the Cook 2013 traffic-light reporting.
- Return type:
Karoly et al. 2017 (Brain) replica metrics.
Paper: ‘The circadian profile of epilepsy improves seizure forecasting’. Reports: AUROC + Brier score + reliability + improvement-over-circadian baseline. Per-window evaluation; SPH 30–60 min.
Citation: Karoly PJ et al., Brain 2017; 140: 2169–2182. doi:10.1093/brain/awx173
- scitex_seizure_metrics.papers.karoly2017.metrics(*, y_true, y_proba, times_seconds=None, seizure_times=None, sph_seconds=1800, sop_seconds=1800, n_surrogate=500, name='karoly2017')[source]¶
Reproduce the metric panel from Karoly 2017.
- Parameters:
y_true – per-window binary labels.
y_proba – per-window predicted probabilities.
times_seconds – optional per-window timestamps; required if forecasting metrics are wanted (alarm-based).
seizure_times – optional seizure onset timestamps; required if forecasting metrics are wanted.
sph_seconds (float) – prediction horizon (Karoly: 30–60 min).
sop_seconds (float) – occurrence period.
n_surrogate (int) – surrogates for IoC.
name (str) – identifier carried into the report.
- Returns:
auroc, brier, reliability, resolution, ece, alarm_sensitivity (if forecasting inputs given), ioc_vs_circadian, ioc_vs_poisson.
- Return type:
dict with keys
Kuhlmann et al. 2018 (Brain) — Epilepsy Ecosystem replica metrics.
Paper: ‘Epilepsyecosystem.org: crowd-sourcing reproducible seizure prediction with long-term human intracranial EEG’. Reports clip-level AUROC (10-min windows preictal vs interictal); pseudo-prospective sensitivity vs proportion-time-in-warning.
Citation: Kuhlmann L et al., Brain 2018; 141: 2619–2630. doi:10.1093/brain/awy210
- scitex_seizure_metrics.papers.kuhlmann2018.metrics(*, y_true, y_proba, name='kuhlmann2018')[source]¶
Reproduce the Kaggle-style metric panel from Kuhlmann 2018.
Maturana et al. 2020 (Nat Commun) replica metrics.
Paper: ‘Critical slowing down as a biomarker for seizure susceptibility’. Reports: per-patient AUROC, IoC, time-in-warning. Long-window (hours- to-days) features, causal filtering.
Citation: Maturana MI et al., Nat Commun 2020; 11: 2172. doi:10.1038/s41467-020-15908-3
- scitex_seizure_metrics.papers.maturana2020.metrics(*, y_true, y_proba, times_seconds=None, seizure_times=None, sph_seconds=0.0, sop_seconds=3600, n_surrogate=500, name='maturana2020')[source]¶
Reproduce the metric panel from Maturana 2020.
Differs from Karoly 2017: shorter SPH (≈ 0, immediate forecast), longer SOP (1 h default), Poisson surrogate (no circadian baseline in the paper).
Proix et al. 2021 (Lancet Neurology) replica metrics.
Paper: ‘Forecasting seizure risk in adults with focal epilepsy: a development and validation study’. Multi-day-horizon forecasting using multidien IEA cycles on RNS data. Reports: - IoC (Improvement-over-Chance) at horizons 1 h, 1 d, 3 d. - AUC of sensitivity vs proportion-time-in-warning. - Brier skill score.
Citation: Proix T et al., Lancet Neurol 2021; 20: 127–135. doi:10.1016/S1474-4422(20)30396-3
- scitex_seizure_metrics.papers.proix2021.metrics(*, y_proba, times_seconds, seizure_times, horizons_seconds=None, n_surrogate=500, name='proix2021')[source]¶
Reproduce the multi-horizon IoC panel.
- Parameters:
- Returns:
sensitivity, ioc, fp_per_hour, time_in_warning_frac. Plus a multidien_baseline IoC vs 7-day-period surrogate.
- Return type:
dict with per-horizon
Stirling et al. 2021 (Frontiers in Neurology) replica metrics.
Paper: ‘Seizure Forecasting Using a Novel Sub-Scalp Ultra-Long Term EEG Monitoring System’. Reports detection AUC + cycle-based forecasting IoC + time-in-warning on a 4-channel sub-scalp device cohort.
Citation: Stirling RE et al., Front Neurol 2021; 12: 713794. doi:10.3389/fneur.2021.713794
- scitex_seizure_metrics.papers.stirling2021.metrics(*, y_true, y_proba, times_seconds=None, seizure_times=None, sph_seconds=3600, sop_seconds=3600, n_surrogate=500, name='stirling2021')[source]¶
Reproduce the detection-AUC + forecasting-IoC panel.
- Parameters:
y_true – per-window binary detection labels.
y_proba – per-window predictions.
times_seconds – timestamps (optional; for forecasting metrics).
seizure_times – seizure onsets (optional).
sop_seconds (float) – forecasting horizon (default 1h/1h).
n_surrogate (int) – surrogate count.
name (str) – identifier.
sph_seconds (float)
sop_seconds
- Returns:
dict with detection_auc, alarm_sensitivity (if forecasting inputs given), ioc, fp_per_hour, time_in_warning_frac.
- Return type:
Andrade et al. 2024 (Front Neurosci) replica metrics.
Paper: ‘On the performance of seizure prediction machine learning methods across different databases: the sample and alarm-based perspectives’. Reports both regimes side-by-side and a surrogate- predictor chance baseline.
Citation: Andrade I, Teixeira C, Pinto M; Front Neurosci 2024; doi:10.3389/fnins.2024.1417748