Metadata-Version: 2.4
Name: temporalcv
Version: 2.3.0
Summary: Temporal cross-validation with leakage protection for time-series ML
Project-URL: Homepage, https://github.com/brandon-behring/temporalcv
Project-URL: Documentation, https://github.com/brandon-behring/temporalcv#readme
Project-URL: Repository, https://github.com/brandon-behring/temporalcv
Project-URL: Issues, https://github.com/brandon-behring/temporalcv/issues
Author: Brandon Behring
License-Expression: MIT
License-File: LICENSE
Keywords: cross-validation,forecasting,leakage-detection,machine-learning,temporal-validation,time-series,walk-forward
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: matplotlib<4.0,>=3.5
Requires-Dist: numpy<3.0,>=1.21
Requires-Dist: scikit-learn<2.0,>=1.0
Requires-Dist: scipy<2.0,>=1.7
Requires-Dist: statsmodels<1.0,>=0.13
Provides-Extra: all
Requires-Dist: datasetsforecast>=0.0.8; extra == 'all'
Requires-Dist: fredapi>=0.5; extra == 'all'
Requires-Dist: gluonts>=0.13; extra == 'all'
Requires-Dist: pandas>=1.3; extra == 'all'
Requires-Dist: statsforecast>=1.5; extra == 'all'
Provides-Extra: benchmarks
Requires-Dist: datasetsforecast>=0.0.8; extra == 'benchmarks'
Requires-Dist: fredapi>=0.5; extra == 'benchmarks'
Provides-Extra: changepoint
Requires-Dist: ruptures>=1.1; extra == 'changepoint'
Provides-Extra: compare
Requires-Dist: statsforecast>=1.5; extra == 'compare'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: ipykernel>=6.0; extra == 'dev'
Requires-Dist: mypy<2.0,>=1.0; extra == 'dev'
Requires-Dist: nbval>=0.11.0; extra == 'dev'
Requires-Dist: pip-audit>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pytest-benchmark>=4.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff<0.16,>=0.15.16; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo>=2024.1; extra == 'docs'
Requires-Dist: matplotlib>=3.5; extra == 'docs'
Requires-Dist: myst-parser>=2.0.0; extra == 'docs'
Requires-Dist: pillow>=9.0; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints>=1.25; extra == 'docs'
Requires-Dist: sphinx-copybutton>=0.5.2; extra == 'docs'
Requires-Dist: sphinx-design>=0.5.0; extra == 'docs'
Requires-Dist: sphinx-gallery>=0.15.0; extra == 'docs'
Requires-Dist: sphinx>=7.2; extra == 'docs'
Requires-Dist: sphinxcontrib-mermaid>=0.9.2; extra == 'docs'
Provides-Extra: fred
Requires-Dist: fredapi>=0.5; extra == 'fred'
Provides-Extra: gluonts
Requires-Dist: gluonts>=0.13; extra == 'gluonts'
Provides-Extra: monash
Requires-Dist: datasetsforecast>=0.0.8; extra == 'monash'
Provides-Extra: pandas
Requires-Dist: pandas>=1.3; extra == 'pandas'
Description-Content-Type: text/markdown

<p align="center">
  <img src="docs/images/logo.svg" alt="temporalcv" width="320">
</p>

<p align="center">
  <strong>Rigorous cross-validation for time series with leakage detection and gap enforcement.</strong>
</p>

<p align="center">
  <a href="https://github.com/brandon-behring/temporalcv/actions/workflows/ci.yml"><img src="https://github.com/brandon-behring/temporalcv/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/temporalcv/"><img src="https://img.shields.io/pypi/v/temporalcv.svg" alt="PyPI"></a>
  <a href="https://temporalcv.readthedocs.io"><img src="https://readthedocs.org/projects/temporalcv/badge/?version=latest" alt="Docs"></a>
  <a href="https://pypi.org/project/temporalcv/"><img src="https://img.shields.io/pypi/pyversions/temporalcv.svg" alt="Python"></a>
  <a href="https://codecov.io/gh/brandon-behring/temporalcv"><img src="https://codecov.io/gh/brandon-behring/temporalcv/branch/main/graph/badge.svg" alt="Coverage"></a>
  <a href="https://colab.research.google.com/github/brandon-behring/temporalcv/blob/main/notebooks/demo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
</p>

<p align="center">
  <a href="https://temporalcv.readthedocs.io">Documentation</a> •
  <a href="https://temporalcv.readthedocs.io/en/latest/auto_examples/">Examples</a> •
  <a href="https://temporalcv.readthedocs.io/en/latest/tutorials/leakage_detection.html">Leakage Tutorial</a>
</p>

---

## Installation

```bash
pip install temporalcv
```

## Quick Start

Validate your model for temporal leakage in 4 lines:

```python
from temporalcv import run_gates
from temporalcv.gates import gate_signal_verification

report = run_gates([gate_signal_verification(model, X, y, n_shuffles=100)])
print(report.status)  # HALT, WARN, or PASS
```

| Status | Meaning |
|--------|---------|
| **HALT** | Signal detected — investigate (legitimate temporal pattern or leakage?) |
| **WARN** | Marginal signal — proceed with caution |
| **PASS** | No signal — model has no detectable predictive power |

---

## Why temporalcv?

Standard cross-validation shuffles data randomly. For time series, this means training on future data to predict the past — a form of data leakage that inflates metrics.

temporalcv provides:
- **Leakage detection** via validation gates (shuffled target test, suspicious improvement)
- **Gap enforcement** for h-step forecasting (no lag-feature contamination)
- **High-persistence metrics** (MASE, MC-SS) that measure actual skill
- **Conformal coverage** with caveats: marginal under exchangeability; time-series autocorrelation can invalidate the guarantee — use `AdaptiveConformalPredictor` (Gibbs & Candès 2021) for distribution-shift handling

---

## Feature Comparison

| Feature | temporalcv | sklearn | sktime | darts |
|---------|:----------:|:-------:|:------:|:-----:|
| Horizon-derived gap (auto) | ✓ | Manual¹ | Manual | Manual |
| Leakage detection gates | ✓ | ✗ | ✗ | ✗ |
| Conformal prediction | ✓ | ✗ | Partial | ✓ |
| sklearn-compatible API | ✓ | ✓ | ✓ | ✗ |
| Statistical tests (DM, PT) | ✓ | ✗ | Partial | ✗ |

¹ sklearn `TimeSeriesSplit(gap=N)` (since v0.24) requires the user to compute `N` from the forecast horizon themselves; temporalcv derives `gap = horizon + extra_gap` automatically from the `horizon` parameter.

---

## Core Features

### Walk-Forward CV with Gap

```python
from temporalcv import WalkForwardCV

cv = WalkForwardCV(
    window_type="sliding",
    window_size=104,    # 2 years of weekly data
    horizon=2,          # 2-step ahead forecast
    test_size=1
)

for train_idx, test_idx in cv.split(X, y):
    model.fit(X[train_idx], y[train_idx])
    pred = model.predict(X[test_idx])
```

The `horizon` parameter enforces a gap between training and test sets, preventing lagged features from leaking target information.

### Validation Gates

```python
from temporalcv.gates import gate_signal_verification, gate_suspicious_improvement

gates = [
    gate_signal_verification(model, X, y, n_shuffles=100),
    gate_suspicious_improvement(model_mae, baseline_mae, threshold=0.20),
]

report = run_gates(gates)
if report.status == "HALT":
    raise ValueError(f"Signal detected — investigate: {report.summary()}")
```

### High-Persistence Metrics

When your series is "sticky" (ACF(1) > 0.9), standard MAE lies — predicting "same as yesterday" looks great but adds no value.

```python
from temporalcv.metrics import mase, mc_skill_score

print(f"MASE: {mase(actual, predicted, y_train):.3f}")
print(f"MC-SS: {mc_skill_score(actual, predicted):.3f}")
```

| Metric | What It Measures |
|--------|------------------|
| **MASE** | Error relative to naive forecast (scale-free) |
| **MC-SS** | Skill only when target moved |

---

## Validation Pipeline

```mermaid
flowchart TD
    A[Data + Model] --> B{Validation Gates}
    B -->|HALT| C[Stop & Investigate]
    B -->|WARN| D[Proceed with Caution]
    B -->|PASS| E[Walk-Forward CV]
    E --> F[Statistical Tests]
    F --> G[Deploy]
```

---

## Common Leakage Patterns

Learn from these failure modes:

| Pattern | Example | Why It's Bad |
|---------|---------|--------------|
| Rolling stats on full series | `.rolling().mean()` without `.shift()` | Features encode future |
| No gap for h-step forecast | `horizon=0` when predicting 2 steps ahead | Lag features leak target |
| Threshold on full data | Regime boundary uses all data | Classification cheats |

See the [failure examples gallery](examples/) for detailed walkthroughs:
- [16: Rolling Stats Leak](examples/16_failure_rolling_stats.py)
- [17: Threshold Leak](examples/17_failure_threshold_leak.py)
- [19: Missing Gap](examples/19_failure_missing_gap.py)
- [20: KFold Trap](examples/20_failure_kfold.py) — 47.8% fake improvement

---

## Optional Dependencies

```bash
pip install temporalcv[benchmarks]   # M4/M5 benchmarks
pip install temporalcv[changepoint]  # PELT algorithm
pip install temporalcv[dev]          # Testing, linting
pip install temporalcv[all]          # Everything
```

**Core**: numpy >= 1.21, scipy >= 1.7, scikit-learn >= 1.0, statsmodels >= 0.13, matplotlib >= 3.5  •  **Optional**: pandas >= 1.3 (`pip install temporalcv[pandas]`)

**Platforms**: Linux, macOS, Windows | **Python**: 3.10+

---

## Documentation

| Resource | Description |
|----------|-------------|
| [Quickstart](https://temporalcv.readthedocs.io/en/latest/quickstart.html) | Get running in 5 minutes |
| [Leakage Tutorial](https://temporalcv.readthedocs.io/en/latest/tutorials/leakage_detection.html) | Deep dive on detection |
| [API Reference](https://temporalcv.readthedocs.io/en/latest/api/) | Full API docs |
| [Examples Gallery](https://temporalcv.readthedocs.io/en/latest/auto_examples/) | 21 real-world cases |

---

## Validation Evidence

| Test | Reference | Result |
|------|-----------|--------|
| DM test golden values | R `forecast::dm.test()` | ✓ Match |
| Type I error rate | 500 Monte Carlo sims | 5% ± 2% |
| Conformal coverage | Synthetic AR(1) | 95% nominal |
| Benchmark | M4 Competition (4,773 series) | ✓ Validated |

See [Testing Strategy](https://temporalcv.readthedocs.io/en/latest/testing_strategy.html) for details.

---

## Citation

```bibtex
@software{temporalcv2025,
  author    = {Behring, Brandon},
  title     = {temporalcv: Temporal cross-validation with leakage protection},
  year      = {2025},
  publisher = {GitHub},
  url       = {https://github.com/brandon-behring/temporalcv}
}
```

---

## Ecosystem

Part of the **Rigorous AI Engineering** ecosystem:

| Project | Description |
|---------|-------------|
| **temporalcv** (this repo) | Temporal cross-validation with leakage detection |
| [ir-eval](https://github.com/brandon-behring/ir-eval) | Statistical retrieval evaluation with drift detection |
| [research-kb](https://github.com/brandon-behring/research-kb) | Graph-boosted semantic search for research literature |

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.

## License

MIT License — see [LICENSE](LICENSE)
