Metadata-Version: 2.4
Name: pitcp
Version: 0.12.1
Summary: PIT-CP conformal prediction with pivotal scores
Author: Félix Laplante
License-Expression: GPL-3.0-only
Project-URL: Source, https://github.com/felixlaplante0/pitcp
Project-URL: Documentation, https://pitcp.readthedocs.io/en/latest/
Project-URL: Playground, https://pitcp-app.streamlit.app/
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: torch
Requires-Dist: scikit-learn
Requires-Dist: zuko
Requires-Dist: tqdm
Provides-Extra: test
Requires-Dist: catboost; extra == "test"
Requires-Dist: lightgbm; extra == "test"
Requires-Dist: pytest<10,>=8; extra == "test"
Requires-Dist: pytest-cov<8,>=5; extra == "test"
Requires-Dist: xgboost; extra == "test"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/felixlaplante0/pitcp/main/docs/source/_static/pitcp-logo.svg" alt="PIT-CP logo" width="128">
</p>

<h1 align="center">PIT-CP</h1>

<p align="center"><strong>Prediction regions that follow the data.</strong><br>
Adaptive conformal prediction with pivotal scores and conditional coverage.</p>

<p align="center">
  <a href="https://pitcp.readthedocs.io/en/latest/">Documentation</a> ·
  <a href="https://pitcp-app.streamlit.app/">Interactive playground</a> ·
  <a href="https://pypi.org/project/pitcp/">PyPI</a>
</p>

<p align="center">
  <a href="https://pypi.org/project/pitcp/"><img src="https://img.shields.io/pypi/v/pitcp?logo=pypi&logoColor=white" alt="PyPI version"></a>
  <a href="https://pypi.org/project/pitcp/"><img src="https://img.shields.io/badge/python-3.11--3.14-blue?logo=python&logoColor=white" alt="Supported Python versions: 3.11–3.14"></a>
  <a href="https://github.com/felixlaplante0/pitcp/actions/workflows/lint.yml"><img src="https://github.com/felixlaplante0/pitcp/actions/workflows/lint.yml/badge.svg" alt="Lint status"></a>
  <a href="https://codecov.io/gh/felixlaplante0/pitcp"><img src="https://codecov.io/gh/felixlaplante0/pitcp/graph/badge.svg" alt="Coverage"></a>
  <a href="https://pitcp.readthedocs.io/en/latest/"><img src="https://readthedocs.org/projects/pitcp/badge/?version=latest" alt="Documentation status"></a>
  <a href="https://github.com/felixlaplante0/pitcp/blob/main/LICENSE"><img src="https://img.shields.io/github/license/felixlaplante0/pitcp" alt="License"></a>
</p>

**pitcp** is a Python package for conformal prediction using **probability integral transform (PIT) pivotal scores**. Given any black-box nonconformity score, it fits a conditional density estimator on the score distribution and maps raw scores to PIT values, yielding valid marginal coverage at any user-specified level.

Our contribution is `PITCP`. The package also reimplements the state-of-the-art `SCP`, `CQR`, `HPD`, and `CONTRA` baselines behind a consistent scikit-learn API.

---

## ✨ Features

- **PIT conformal prediction**: `PITCP` maps base nonconformity scores through a learned conditional CDF.
- **Split conformal prediction**: `SCP` calibrates arbitrary scalar nonconformity scores without a learned correction.
- **Conformalized quantile regression**: `CQR` accepts multiple outputs and provides a scikit-learn gradient-boosting implementation of state-of-the-art conformalized quantile regression.
- **Highest-density regions**: `HPD` calibrates conditional highest-predictive-density sets.
- **Latent-space regions**: `CONTRA` maps targets through a conditional normalizing flow and calibrates a Euclidean norm-based score in latent space.
- **Model-agnostic**: Works with any callable nonconformity score `s(x, y)`, including distance, residual, or likelihood-based scores.
- **Flexible Density Estimation**: Supports normalizing flows and mixture density networks from the [zuko](https://github.com/probabilists/zuko) library.
- **Marginal Coverage Guarantee**: Provably valid conformal coverage at any target level via finite-sample calibration.
- **scikit-learn integration**: Native `BaseEstimator` integration with familiar `fit`, `conformalize`, `predict`, and `contains` methods.

---

## 🚀 Installation

```bash
python -m pip install pitcp
```

## 🔧 Usage

### Example

```python
import torch
import zuko
from pitcp import PITCP


def std(x):
    return torch.where((x > -0.9) & (x < 0.9), torch.cos(torch.pi * x / 2), 1.0)


def gen_data(n):
    x = torch.rand(n, 1) * 2 - 1
    return x, torch.randn(n, 1) * std(x)


torch.manual_seed(42)

(X_train, y_train), (X_cal, y_cal), (X_test, y_test) = [
    gen_data(5000) for _ in range(3)
]


# Define a nonconformity score
def score(x, y):
    return y.abs()


# Build a normalizing flow density estimator
model = zuko.flows.NSF(features=1, context=1, bins=4, hidden_features=(32, 32, 32))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)

# Compute nonconformity scores
s_train = score(X_train, y_train)
s_cal = score(X_cal, y_cal)
s_test = score(X_test, y_test)

# Fit and conformalize
pitcp = PITCP(model, optimizer, n_epochs=10, batch_size=128)
pitcp.fit(X_train, s_train)
pitcp.conformalize(X_cal, s_cal)

# Predict conformal regions (max score thresholds) at multiple quantiles
limits = pitcp.predict(X_test, confidence_level=[0.7, 0.8, 0.9])

# Predict conformal coverage
covered = pitcp.contains(X_test, s_test, confidence_level=[0.7, 0.8, 0.9])
print(f"Empirical coverages: {covered.mean(axis=0)}")
```

---

## 📖 Learn More

For tutorials and the API reference, visit the
[Read the Docs documentation](https://pitcp.readthedocs.io/en/latest/). Try the methods in the
[interactive playground](https://pitcp-app.streamlit.app/).

The method is described in [*A Post-Processing Conformal Prediction Approach
for Conditional Coverage via Pivotal Scores*](https://doi.org/10.48550/arXiv.2605.25852).

## 📚 References

- **Split conformal prediction:** Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. “Distribution-Free Predictive Inference for Regression.” *Journal of the American Statistical Association*, 113(523), 1094–1111, 2018. [DOI](https://doi.org/10.1080/01621459.2017.1307116).
- **Conformalized quantile regression:** Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. “Conformalized Quantile Regression.” *Advances in Neural Information Processing Systems 32*, 2019. [NeurIPS](https://papers.neurips.cc/paper_files/paper/2019/hash/5103c3584b063c431bd1268e9b5e76fb-Abstract.html).
- **HPD-split:** Rafael Izbicki, Gilson Shimizu, and Rafael B. Stern. “CD-split and HPD-split: Efficient Conformal Regions in High Dimensions.” *Journal of Machine Learning Research*, 23(87), 1–32, 2022. [JMLR](https://www.jmlr.org/papers/v23/20-797.html).
- **CONTRA:** Zhenhan Fang, Aixin Tan, and Jian Huang. “CONTRA: Conformal Prediction Region via Normalizing Flow Transformation.” *International Conference on Learning Representations*, 2025. [ICLR](https://proceedings.iclr.cc/paper_files/paper/2025/hash/e55d081280e79e714debf2902e18eb69-Abstract-Conference.html).

---

## 📊 Reproducing Results

Clone the repository, create and activate a virtual environment, then install the exact package versions used for the paper:

```bash
python -m pip install -r scripts/requirements.txt
```

Keep the repository folder layout unchanged: do not move, rename, or flatten its folders, because the experiment scripts resolve paths relative to their file locations.

This installs the frozen `pitcp` release from PyPI together with the
experimental dependencies recorded in `scripts/requirements.txt`.
Do not subsequently install the repository in editable mode when reproducing
the paper, because that would replace the frozen PyPI release with the local
source checkout.

For local development rather than exact paper reproduction, install the current checkout in editable mode from the repository root:

```bash
python -m pip install -e .
```

The editable installation makes local source changes immediately available without reinstalling the package.

### Preparing the Real Data

The repository contains the raw SARCOS (`data/sarcos_inv.mat`) and Naval Propulsion Plants (`data/naval.txt`) datasets. To regenerate the training, validation-test, and prediction CSV files, install the TabPFN client, set the `TABPFN_ACCESS_TOKEN` environment variable to a valid access token, and run both dataset modes:

```bash
python -m pip install tabpfn-client
```

On PowerShell:

```powershell
$env:TABPFN_ACCESS_TOKEN="your-token"
python data/predict.py --sarcos
python data/predict.py --naval
```

On Linux or macOS:

```bash
export TABPFN_ACCESS_TOKEN="your-token"
python data/predict.py --sarcos
python data/predict.py --naval
```

The token remains in the current terminal session and is not written to the
repository.

These commands write `{dataset}-train.csv`, `{dataset}-valtest.csv`, and `{dataset}-pred.csv` to `data/`. The generated CSV files are already included, so this step can be skipped unless the predictions must be regenerated.

The paper experiments use Python 3.13 and the dependencies in
`scripts/requirements.txt`. Verify downloaded and generated data against
`data/SHA256SUMS` before running the experiments. The committed prediction
files are the canonical reproduction artifacts because results returned by the
remote TabPFN service may change independently of this repository.

On systems with `sha256sum`, verify the artifacts with:

```bash
cd data
sha256sum --check SHA256SUMS
```

### Dataset Attribution and Licensing

The SARCOS inverse-dynamics dataset was provided by Sethu Vijayakumar and
is distributed through the [Gaussian Processes for Machine Learning dataset
repository](https://gaussianprocess.org/gpml/data/). Users should cite
Vijayakumar and Schaal (2000), *LWPR: An O(n) Algorithm for Incremental Real
Time Learning in High Dimensional Space*. No explicit redistribution license
has been identified for SARCOS. Its inclusion in this repository should not be
interpreted as granting reuse or redistribution rights.

The Naval Propulsion Plants dataset is licensed under the
[Creative Commons Attribution 4.0 International license](https://creativecommons.org/licenses/by/4.0/).
Users should cite Coraddu et al. (2014), *Condition Based Maintenance of Naval
Propulsion Plants*, [UCI Machine Learning Repository](https://doi.org/10.24432/C5K31K).

### Running the Experiments

Run the experiment scripts from the repository root:

```bash
python scripts/convergence-plots.py
python scripts/synthetic-plots.py
python scripts/real-data-diagnostics.py --sarcos
python scripts/real-data-diagnostics.py --naval
```

Alternatively, run all four experiments sequentially on Windows, Linux, or macOS:

```bash
python run-all.py
```

The runner uses the active Python interpreter and stops if an experiment fails. It does not run `data/predict.py`; prepare or verify the real-data files separately as described above.

These commands write figures and diagnostic tables to `figures/`. The paper settings are defined as constants near the top of each script.

### Script Descriptions

| Script | Description |
| :--- | :--- |
| `convergence-plots.py` | Evaluates the convergence of the PIT-CP procedure across different training sample sizes using various density estimators (SOSPF, GMM). |
| `real-data-diagnostics.py` | Benchmarks `PITCP` against other conformal prediction methods (`SCP`, `CQR`, `HPD`, `CONTRA`) on real-world datasets, calculating coverage gaps and prediction interval volumes. |
| `synthetic-plots.py` | Compares the conformal regions and conditional coverage of `PITCP`, `CQR`, and `SCP` on synthetic heteroscedastic data. |
