Metadata-Version: 2.4
Name: conformalpy
Version: 1.0.0
Summary: Python library for conformal prediction.
Author: Antonio Caparrini, Javier Arroyo
Maintainer-email: Antonio Caparrini <acaparri@ucm.es>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Caparrini/conformalpy
Project-URL: Source, https://github.com/Caparrini/conformalpy
Project-URL: Issues, https://github.com/Caparrini/conformalpy/issues
Project-URL: Documentation, https://caparrini.github.io/conformalpy/
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21
Requires-Dist: scikit-learn>=1.0
Requires-Dist: matplotlib>=3.5
Requires-Dist: seaborn>=0.12
Requires-Dist: scipy>=1.9
Requires-Dist: pandas>=1.5
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Provides-Extra: docs
Requires-Dist: quartodoc>=0.9; extra == "docs"
Requires-Dist: griffe<1.6,>=1.0; extra == "docs"
Requires-Dist: jupyter>=1.0; extra == "docs"
Provides-Extra: explainability
Requires-Dist: shap>=0.42; extra == "explainability"
Provides-Extra: gpu
Requires-Dist: cupy-cuda12x>=12.0; extra == "gpu"
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Provides-Extra: keras
Requires-Dist: tensorflow>=2.12; extra == "keras"
Provides-Extra: deep-learning
Requires-Dist: conformalpy[keras,torch]; extra == "deep-learning"
Provides-Extra: mlflow
Requires-Dist: mlflow>=2.0; extra == "mlflow"
Provides-Extra: all
Requires-Dist: conformalpy[dev,docs,mlflow]; extra == "all"
Dynamic: license-file

# conformalpy

[![codecov](https://codecov.io/github/Caparrini/conformalpy/graph/badge.svg?token=L2X3B66NTQ)](https://codecov.io/github/Caparrini/conformalpy)
[![CI](https://github.com/caparrini/conformalpy/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/caparrini/conformalpy/actions/workflows/CI.yml)
[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://caparrini.github.io/conformalpy/)
![PyPI - Version](https://img.shields.io/pypi/v/conformalpy)
![Python Version](https://img.shields.io/pypi/pyversions/conformalpy)
[![Shield: Buy me a coffee](https://img.shields.io/badge/Buy%20me%20a%20coffee-Support-yellow?logo=buymeacoffee)](https://www.buymeacoffee.com/caparrini)

A Python library for **Conformal Prediction** with a focus on operational deployment, visualization, and explainability.

## Features

- **Classification**: Binary and multiclass with ICP, CV+, and Mondrian methods
- **Regression**: Split conformal, CQR, Normalized, and Jackknife+ methods
- **Runtime Alpha**: Change confidence level at prediction time without recalibration
- **Operational Framework**: Outcome categorization (SC/SI/TS0/TS1 for binary; SC/SI/MC/MU for multiclass), decision rules, cost-benefit analysis
- **Rich Visualization**: Coverage analysis, prediction intervals, FCODs (Feature-Conditioned Outcome Distributions)
- **sklearn Compatible**: Follows scikit-learn API conventions

## Installation

```bash
pip install conformalpy
```

For development:

```bash
pip install conformalpy[dev]
```

## Quick Start

### Classification

```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

from conformalpy.classifier import ConformalClassifier
from conformalpy.nonconformity.classification import lac_nonconformity

# Load data and split
X, y = load_iris(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)
X_calib, X_test, y_calib, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Train your model
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# Create conformal classifier
conformal = ConformalClassifier(
    model=model,
    alpha=0.1,  # 90% coverage target
    nonconformity_function=lac_nonconformity
)

# Calibrate on held-out data
conformal.calibrate(X_calib, y_calib)

# Get prediction sets
prediction_sets = conformal.predict(X_test)
# Example output: [[0], [1, 2], [2], ...]

# Runtime alpha: get sets at different confidence levels without recalibration
sets_95 = conformal.predict(X_test, alpha=0.05)  # 95% coverage
sets_80 = conformal.predict(X_test, alpha=0.20)  # 80% coverage
```

### Regression

```python
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor

from conformalpy.regressor import ConformalRegressor

# Load data
X, y = fetch_california_housing(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)
X_calib, X_test, y_calib, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Train model
model = GradientBoostingRegressor(random_state=42)
model.fit(X_train, y_train)

# Create conformal regressor
conformal = ConformalRegressor(model=model, alpha=0.1)
conformal.calibrate(X_calib, y_calib)

# Get prediction intervals
intervals = conformal.predict_interval(X_test)
# intervals.shape = (n_samples, 2) -> [lower, upper]

# Runtime alpha: multiple confidence levels at once
intervals_multi = conformal.predict_interval(X_test, alpha=[0.05, 0.10, 0.20])
# intervals_multi.shape = (n_samples, 2, 3)
```

## Main Components

### Classification

```python
from conformalpy.classifier import ConformalClassifier
from conformalpy.nonconformity.classification import (
    lac_nonconformity,    # LAC (Least Ambiguous Classifier): smallest sets
    aps_nonconformity,    # APS (Adaptive Prediction Sets): no empty sets
    raps_nonconformity,   # RAPS (Regularized APS): balanced
    saps_nonconformity,   # SAPS (Sorted Adaptive Prediction Sets): sorted variant
    hinge_nonconformity,  # Hinge: margin-based scores
)

# Standard ICP
icp = ConformalClassifier(model, alpha=0.1, nonconformity_function=lac_nonconformity)

# Mondrian (class-conditional coverage)
mondrian = ConformalClassifier(model, alpha=0.1, nonconformity_function=lac_nonconformity, mondrian=True)

# Flexible Mondrian (arbitrary groups)
conformal.calibrate(X_calib, y_calib, groups=group_labels)
conformal.predict(X_test, groups=test_groups)

# CV+ (cross-validation based)
from conformalpy.classifier import CVPlusClassifier
cv_plus = CVPlusClassifier(model, alpha=0.1, n_splits=5)
cv_plus.fit(X_train, y_train)  # No separate calibration needed
```

### Regression

```python
from conformalpy.regressor import ConformalRegressor, CQR, NormalizedConformalRegressor

# Split conformal (symmetric intervals)
split = ConformalRegressor(model, alpha=0.1)

# CQR (adaptive intervals via quantile regression)
cqr = CQR(model_lower=quantile_lower, model_upper=quantile_upper, alpha=0.1)

# Normalized (scaled by predicted uncertainty)
normalized = NormalizedConformalRegressor(model, sigma_model, alpha=0.1)
```

### Evaluation

```python
from conformalpy.evaluation import (
    coverage_score,
    average_set_size,
    class_conditional_coverage,
    interval_coverage_score,
    average_interval_width,
    winkler_score,
)

# Classification metrics
coverage = coverage_score(prediction_sets, y_test)
avg_size = average_set_size(prediction_sets)
class_cov = class_conditional_coverage(prediction_sets, y_test)

# Regression metrics
cov = interval_coverage_score(y_test, intervals)
width = average_interval_width(intervals)
winkler = winkler_score(y_test, intervals, alpha=0.1)
```

### Operational Framework

```python
from conformalpy.outcomes import categorize_outcomes, outcome_summary
from conformalpy.framework import classify_zones, compute_expected_cost

# Binary: SC (correct), SI (incorrect), TS0/TS1 (two-set)
# Multiclass: SC (correct), SI (incorrect), MC (multi-covered), MU (multi-uncovered)
outcomes = categorize_outcomes(prediction_sets, y_test)

# Operational zones: Safe, Uncertain, Problematic
zones = classify_zones(prediction_sets, y_test)

# Cost-benefit analysis
from conformalpy.framework import CostMatrix
costs = CostMatrix(false_negative_cost=1000, false_positive_cost=100, manual_review_cost=50)
y_pred = model.predict(X_test)
cost_report = compute_expected_cost(prediction_sets, y_pred, y_test, costs)  # dict with cost breakdown
```

## Score Function Comparison

| Method | Empty Sets | Set Size | Best For |
|--------|------------|----------|----------|
| LAC | Possible | Smallest | When empty sets are acceptable |
| APS | Never | Larger | When adaptiveness is important |
| RAPS | Never | Small | General use (recommended) |
| SAPS | Never | Small | Sorted variant of APS |
| Hinge | Possible | Smallest | Margin-based scoring |

## Additional Features

### Time Series

Adaptive Conformal Inference (ACI) for sequential data under distribution shift.

```python
from conformalpy.timeseries import AdaptiveConformalRegressor

aci = AdaptiveConformalRegressor(model=model, target_coverage=0.9, gamma=0.01)
```

### Risk Control

Conformal Risk Control (CRC), RCPS, and Learn-then-Test (LTT) for multi-label and custom loss functions.

```python
from conformalpy.risk_control import ConformalRiskController
```

### FCODs (Feature-Conditioned Outcome Distributions)

Visualize how prediction outcomes (SC, SI, MC, MU, etc.) vary as a function of input features — an original contribution of conformalpy.

```python
from conformalpy.fcod import compute_fcod, plot_fcod
```

### Multi-Output Regression

Marginal, Bonferroni-corrected, and max-residual strategies for multi-target regression.

```python
from conformalpy.regressor import MultiOutputConformalRegressor
```

### Jackknife+

Leave-one-out conformal regressor with finite-sample coverage guarantees.

```python
from conformalpy.regressor import JackknifePlusRegressor
```

### Conformal Predictive Distributions

Full predictive distributions (CDFs) via conformal prediction.

```python
from conformalpy.regressor import ConformalPredictiveDistribution
```

### Deep Learning Integration

Adapters for PyTorch and TensorFlow models.

```python
from conformalpy.deep_learning import TorchClassifierWrapper
```

### Diagnostics

Exchangeability testing via martingales and plugin methods.

```python
from conformalpy.diagnostics import ExchangeabilityTest
```

Also available and documented on the [docs site](https://caparrini.github.io/conformalpy/):
weighted / clustered / group-balanced classifiers, fairness metrics, SHAP dependence plots,
model persistence, MLflow integration, and optional GPU backends.

## Citation

The conformalpy library paper is in preparation. If you use the FCOD or
p-value-margin SHAP functionality, please cite:

```bibtex
@inproceedings{caparrini2026explaining,
  author    = {Antonio Caparrini and Miller-Janny Ariza-Garz{\'o}n and Javier Arroyo},
  title     = {Explaining Conformal Prediction: Diagnosing Reliability through Feature-Conditioned Outcomes and p-Value Margins},
  booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
  series    = {Proceedings of Machine Learning Research},
  volume    = {329},
  year      = {2026},
  publisher = {PMLR},
  note      = {Conditionally accepted}
}
```

## Benchmarks

conformalpy achieves equivalent coverage to established libraries while being faster:

| Library | Avg Time (Classification) | Coverage |
|---------|---------------------------|----------|
| conformalpy | **0.04s** | Valid |
| Crepes | 0.11s | Valid |

See `benchmarks/` for full comparison.

## Documentation

Full documentation: [caparrini.github.io/conformalpy](https://caparrini.github.io/conformalpy/)

To build and preview the documentation locally:

```bash
# Install documentation dependencies
uv sync --extra dev --extra docs --extra explainability

# Build the documentation
cd docs && uv run quarto render

# Preview the documentation (in another terminal)
cd docs && uv run quarto preview
```

The documentation uses Quarto with quartodoc for API reference generation. See [docs/_quarto.yml](docs/_quarto.yml) for the full configuration.

## References

**Conformal Prediction:**
- Vovk, V., Gammerman, A., & Shafer, G. (2005). *Algorithmic Learning in a Random World*. Springer.
- Angelopoulos, A. N., & Bates, S. (2023). *Conformal Prediction: A Gentle Introduction*. Foundations and Trends in ML.

**FCOD and SHAP Margins:**
- Caparrini, A., Ariza-Garzón, M.-J., & Arroyo, J. (2026). *Explaining Conformal Prediction: Diagnosing Reliability through Feature-Conditioned Outcomes and p-Value Margins*. COPA, PMLR 329. (Conditionally accepted)

**Score Functions:**
- Sadinle, M., Lei, J., & Wasserman, L. (2019). *Least Ambiguous Set-Valued Classifiers With Bounded Error Levels*. JASA. (LAC)
- Romano, Y., Sesia, M., & Candès, E. J. (2020). *Classification with Valid and Adaptive Coverage*. NeurIPS. (APS)
- Angelopoulos, A. N., et al. (2021). *Uncertainty Sets for Image Classifiers using Conformal Prediction*. ICLR. (RAPS)
- Huang, J., et al. (2024). *Conformal Prediction for Deep Classifier via Label Ranking*. ICML. (SAPS)
- Romano, Y., Patterson, E., & Candès, E. J. (2019). *Conformalized Quantile Regression*. NeurIPS. (CQR)

**Additional Methods:**
- Gibbs, I. & Candès, E. (2021). *Adaptive Conformal Inference Under Distribution Shift*. NeurIPS. (ACI)
- Barber, R. F., et al. (2021). *Predictive Inference with the Jackknife+*. Annals of Statistics.
- Bates, S., et al. (2021). *Distribution-Free, Risk-Controlling Prediction Sets*. Journal of the ACM.

## Contributing

Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow.

```bash
# Clone the repository
git clone https://github.com/Caparrini/conformalpy.git
cd conformalpy

# Install with uv
uv sync --extra dev --extra docs --extra explainability

# Run tests
uv run pytest tests/

# Build API reference from Python docstrings
cd docs && uv run quartodoc build

# Render docs website
cd docs && uv run quarto render

# Preview docs (in another terminal)
cd docs && uv run quarto preview
```

## License

This project is licensed under the [MIT License](https://github.com/Caparrini/conformalpy/blob/main/LICENSE).

See [CHANGELOG.md](CHANGELOG.md) for release history.
