Metadata-Version: 2.4
Name: tinyconformal
Version: 0.7.6
Summary: A small toolbox for conformal prediction
Author-email: Lucas Leão <heylucasleao@gmail.com>
License-Expression: MIT
Keywords: machine-learning,conformal-prediction
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.4
Requires-Dist: venn-abers>=1.4.6
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: quantile-forest>=1.4.2
Provides-Extra: plot
Requires-Dist: tinyshift[plot]>=1.9.0; extra == "plot"
Provides-Extra: notebook
Requires-Dist: nbformat>=5.10.4; extra == "notebook"
Requires-Dist: ipykernel>6.7.0; extra == "notebook"
Provides-Extra: series
Requires-Dist: statsforecast>=1.4.0; extra == "series"
Requires-Dist: mlforecast>=0.7.0; extra == "series"
Requires-Dist: utilsforecast>=0.1.0; extra == "series"
Provides-Extra: all
Requires-Dist: tinyshift[notebook,plot,series]>=1.9.0; extra == "all"
Provides-Extra: dev
Requires-Dist: tinyshift[all]>=1.9.0; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# TinyConformal

Related project: [tinyshift](https://github.com/HeyLucasLeao/tinyshift)

TinyConformal is a Python library for conformal prediction in classification and regression.
It provides tools to build valid prediction sets and prediction intervals with a target significance level (`alpha`).

It also provides conformal predictive systems (CPS) that turn fitted point
regressors into complete predictive distributions for continuous or ordered
discrete outcomes.

For more information on a previous project related to Out-of-Bag (OOB) solutions, visit [this link](https://github.com/HeyLucasLeao/cp-study).

## Recent updates
- Added out-of-fold regression calibration through `CrossValidationCalibration`.
- Added `fit_from_scores` for ICP/CQR and standardized residual calibration for CPS.
- Classifiers and regressors can now reuse out-of-fold cross-validation outputs for conformal calibration without reserving a separate calibration split.
- Added `tinyconformal.series` support with `MultiStepConformalTimeSeriesRegressor ` and `ConformalizedQuantileTimeSeriesRegressor` for multi-step time series interval forecasting with customizable backtesting strides (`step_size`).
- Added support for Conformalized Quantile Regression (CQR) on multi-step time series using base estimators producing quantile forecasts.

Previously, `calibrate` used `Balanced Accuracy Score`; it can now also be calibrated with `Matthews Correlation Coefficient` or `Bookmaker Informedness Score` for improved reliability. The `evaluate` method also reports `bm` and `mcc`.

Currently, TinyConformal supports Out-of-Bag (OOB) solutions for `RandomForestClassifier` in binary classification problems, as well as `RandomForestRegressor` and `RandomForestQuantileRegressor` for regression tasks. For additional options and advanced features, you may want to explore [Crepes](https://github.com/henrikbostrom/crepes).

## Installation

### Using pip

```bash
pip install tinyconformal
```

Optional extras:

```bash
pip install "tinyconformal[plot]"
pip install "tinyconformal[notebook]"
pip install "tinyconformal[dev]"
```

### Using uv

Install in the current environment:

```bash
uv pip install tinyconformal
```

Add as a dependency in a project:

```bash
uv add tinyconformal
```

Optional extras with uv:

```bash
uv pip install "tinyconformal[plot]"
uv pip install "tinyconformal[notebook]"
uv pip install "tinyconformal[dev]"
```

## Submodules and usage

TinyConformal is organized into six public submodules:

- [`tinyconformal.classifier`](tinyconformal/classifier/README.md): conformal
  prediction sets for binary classification.
- [`tinyconformal.regressor`](tinyconformal/regressor/README.md): ICP and CQR
  prediction intervals.
- [`tinyconformal.distribution`](tinyconformal/distribution/README.md):
  cross-fitted conformal predictive distributions.
- [`tinyconformal.series`](tinyconformal/series/README.md): MSCP, TSCQR, and
  complete predictive systems for time series.
- [`tinyconformal.core`](tinyconformal/core/README.md): out-of-fold calibration
  primitives.
- [`tinyconformal.utils`](tinyconformal/utils/README.md): multi-quantile modeling
  and Newsvendor decision utilities.

### Predictive distribution submodule

Cross-fit location and scale estimators on the training data, then request any
quantile, central interval, or CDF value:

```python
from tinyconformal.distribution import ContinuousCrossConformalPredictiveSystem

cps = ContinuousCrossConformalPredictiveSystem(
    learner=location_regressor,
    dispersion_learner=scale_regressor,
)
cps.fit(X_train, y_train, cv=5)
predictive = cps.predict_distribution(X_test)

median = predictive.ppf(0.5)
intervals = predictive.interval(coverage=0.90)
probabilities = predictive.cdf(values)
```

The dispersion learner must return a strictly positive conditional scale
estimate directly (not a variance). CPS internally generates OOF location
residuals, cross-fits the scale learner on their absolute values, standardizes
the residuals, and finally refits both models on all training rows.

For ordered integer targets such as demand counts, use
`DiscreteCrossConformalPredictiveSystem`. Its predictive object additionally exposes
`pmf(values)` and returns integer quantiles. For nominal, unordered labels, use
the classifiers in `tinyconformal.classifier` instead.

The complete distribution can be passed directly to the Newsvendor solver:

```python
from tinyconformal.utils import NewsvendorSolver

result = NewsvendorSolver.optimize_distribution(
    forecast_frame,
    predictive,
    underage_cost="shortage_cost",
    overage_cost="holding_cost",
)
```

For a discrete predictive distribution, the solver can also report the expected
net benefit of adding each inventory unit. The calculation uses the conformal
CDF directly and accepts either ``max_k`` or an explicit unit grid:

```python
marginal_benefit = NewsvendorSolver.marginal_benefit_distribution(
    forecast_frame,
    predictive,
    underage_cost="shortage_cost",
    overage_cost="holding_cost",
    units=[0, 5, 10, 15],
)
```

The tabular CPS supports cross-fitting only; it does not implement split
calibration or CV+. For time series, use the horizon-wise series CPS below.

For Nixtla-compatible estimators, use the horizon-wise series CPS. It shares the
sequential rolling-origin backtesting machinery and panel contract used by MSCP
and TSCQR:

```python
from sklearn.ensemble import RandomForestRegressor

from tinyconformal.series import (
    ContinuousTimeSeriesConformalPredictiveSystem,
)

cps = ContinuousTimeSeriesConformalPredictiveSystem(
    learner=mlforecast_or_statsforecast,
    dispersion_learner=RandomForestRegressor(min_samples_leaf=5),
)
cps.fit(
    train_df,
    horizon=14,
    n_windows=5,
    step_size=14,
    static_features=["store_type"],
    nexcp=True,
    decay=0.99,
    weighted_refit=True,
)

forecast = cps.predict_distribution(h=14, X_df=future_exog)
median_df = forecast.ppf(0.5)
probability_df = forecast.cdf(values)
quantile_df = forecast.ppf([0.1, 0.5, 0.9])
interval_df = forecast.interval(coverage=0.9)
```

The scale estimator is cross-fitted on absolute rolling-origin errors using
series identity and forecast horizon as conditional features. TSCPS requires
the Nixtla learner to contain exactly one model. The returned forecast owns its
point-forecast panel and distribution, so `cdf`, `ppf`, and `interval` return
row-aligned DataFrames. Use
`DiscreteTimeSeriesConformalPredictiveSystem` for ordered integer/count
targets; those distributions additionally provide `pmf`.

MSCP, TSCQR, and TSCPS share the optional NexCP-style temporal weighting
contract. With `nexcp=True` (the default), weights decay exponentially from the newest window
using `decay=0.99`, the value used in the NexCP paper experiments. This weights
calibration scores and, when `weighted_refit=True`, adds an internal recency
weight column to every rolling-origin fit and to the final learner refit. A
learner without `weight_col` support raises an explicit error. TSCPS also passes
window weights to dispersion estimators that support `sample_weight`.

The runnable `examples/distribution/cps.ipynb` notebook covers continuous and
discrete cross-conformal predictive systems in one place.

It covers CDF, SF, PMF where applicable, PPF, arbitrary quantiles, empirical
coverage, and Newsvendor optimization.

The complete OOF workflow for ICP, CQR, CPS, and both binary classification
strategies is available in `examples/calibration/cross_validation.ipynb`.

### Classifier submodule

Import from `tinyconformal.classifier`:

```python
from tinyconformal.classifier import BinaryMarginalConformalClassifier
from tinyconformal.classifier import BinaryClassConditionalConformalClassifier
```

### Regressor submodule

Import from `tinyconformal.regressor`:

```python
from tinyconformal.regressor import ConformalizedRegressor
from tinyconformal.regressor import ConformalizedQuantileRegressor
from tinyconformal.core.calibration import CrossValidationCalibration
```

### Time series submodule

Use the cross-fitted, horizon-wise predictive systems exported by
`tinyconformal.series`:

```python
from tinyconformal.series import (
    MultiStepConformalTimeSeriesRegressor ,
    ConformalizedQuantileTimeSeriesRegressor,
    ContinuousTimeSeriesConformalPredictiveSystem,
    DiscreteTimeSeriesConformalPredictiveSystem,
)
```

### Example

Example usage of `BinaryClassConditionalConformalClassifier`:

```python
from sklearn.ensemble import RandomForestClassifier
from tinyconformal.classifier import BinaryClassConditionalConformalClassifier

# Create and fit a RandomForestClassifier
learner = RandomForestClassifier(n_estimators=100, oob_score=True)
X_train, y_train = ...  # your training data
learner.fit(X_train, y_train)

# Create and fit the conformal classifier
conformal_classifier = BinaryClassConditionalConformalClassifier(learner)
conformal_classifier.fit(y=y_train, oob=True)

# Make predictions
X_test = ...  # your test data
predictions = conformal_classifier.predict(X_test)
```

### Cross-validation calibration example

Use out-of-fold probabilities to calibrate a classifier without reserving a
separate calibration split:

```python
from sklearn.ensemble import RandomForestClassifier
from tinyconformal.core.calibration import CrossValidationCalibration
from tinyconformal.classifier import BinaryMarginalConformalClassifier

learner = RandomForestClassifier(n_estimators=100, oob_score=True)
probabilities = CrossValidationCalibration.classification_probabilities(
    learner, X_train, y_train, cv=5
)
learner.fit(X_train, y_train)

conformal_classifier = BinaryMarginalConformalClassifier(learner)
conformal_classifier.fit_from_probabilities(probabilities, y_train)

predictions = conformal_classifier.predict(X_test)
```

For regressors, generate out-of-fold scores and then fit the final learner on all
available training data:

```python
from sklearn.ensemble import RandomForestRegressor
from tinyconformal.core.calibration import CrossValidationCalibration
from tinyconformal.regressor import ConformalizedRegressor

learner = RandomForestRegressor(random_state=42)
scores = CrossValidationCalibration.icp_scores(
    learner, X_train, y_train, cv=5
)

# Fit learner before using conformal regressor
learner.fit(X_train, y_train)

regressor = ConformalizedRegressor(learner, alpha=0.05)
regressor.fit_from_scores(scores)

intervals = regressor.predict_interval(X_test)
```

### Evaluating the Classifier

Evaluate the performance of the conformal classifier using the `evaluate` method:

```python
results = conformal_classifier.evaluate(X_test, y_test)
print(results)
```

### Time Series Example

`ContinuousTimeSeriesConformalPredictiveSystem` cross-fits a location forecaster
and a conditional-scale model over rolling-origin windows. It returns complete,
horizon-specific predictive distributions for Nixtla-style learners
(MLForecast or StatsForecast):

```python
from lightgbm import LGBMRegressor
from mlforecast import MLForecast
from tinyconformal.series import ContinuousTimeSeriesConformalPredictiveSystem

# Wrap a base forecaster
mlf = MLForecast(
    models=[LGBMRegressor(random_state=42)],
    freq="D",
    lags=[1, 7],
)

conformal_ts = ContinuousTimeSeriesConformalPredictiveSystem(
    learner=mlf,
    dispersion_learner=LGBMRegressor(random_state=42),
)

conformal_ts.fit(df, horizon=7, n_windows=5, step_size=7)
forecast = conformal_ts.predict_distribution(h=7)
intervals_df = conformal_ts.predict_interval(h=7)
```

For ordered integer targets such as demand, use the discrete cross-fitted system.
Its `minimum` is the lower support bound: keep `0` for counts, use `1` for
strictly positive quantities, or `None` if negative integers are valid:

```python
from lightgbm import LGBMRegressor
from mlforecast import MLForecast
from tinyconformal.series import DiscreteTimeSeriesConformalPredictiveSystem

mlf = MLForecast(
    models=[LGBMRegressor(random_state=42)],
    freq="D",
    lags=[1, 7],
)

conformal_count_ts = DiscreteTimeSeriesConformalPredictiveSystem(
    learner=mlf,
    dispersion_learner=LGBMRegressor(random_state=42),
    minimum=0,
)

conformal_count_ts.fit(df, horizon=7, n_windows=5, step_size=7)
forecast = conformal_count_ts.predict_distribution(h=7)
```

#### Future features and evaluation data

Columns passed through `static_features` belong to each series and are supplied to
the learner only during fitting. All other non-structural columns in the training
data are treated as dynamic exogenous features and must be available for future
timestamps through `X_df`:

```python
conformal_ts.fit(
    train_df,
    static_features=["region"],
)
intervals_df = conformal_ts.predict_interval(
    h=7,
    X_df=future_df[["unique_id", "ds", "temperature"]],
)
```

An explicit `X_df` must contain the identifier, time, and every dynamic exogenous
column used during fitting. It must also contain exactly `h` unique timestamps per
series, using the same timestamp grid for every series. The prediction horizon must
be positive and cannot exceed the `horizon` used for calibration.

`evaluate(df_test, h=...)` uses dynamic features from `df_test` and requires exactly
one non-missing target for every predicted identifier/timestamp pair. Duplicate or
missing targets raise an error instead of being silently omitted from the metrics.

MSCP supports fractional coverage levels. For example, `alpha=0.055` produces
columns such as `Model-lo-94.5` and `Model-hi-94.5`.

Finite-sample conformal correction uses discrete order statistics. When the
requested coverage cannot be attained with the available calibration sample, a
`RuntimeWarning` is emitted and the rank is clipped to the observed score range.
Increasing the number of calibration trajectories, usually through more windows or
series, permits more extreme coverage levels.

### Time Series Mechanics: Horizon vs. Step Size

When calibrating over time series, nonconformity scores are extracted by performing sequential backtesting across multiple calibration windows. The calibration movement is controlled by two parameters:
- `horizon` ($H$): The forecast horizon step count generated in each window.
- `step_size` ($S$): The stride length used to advance the origin between backtesting windows.

Below are three typical backtesting movement patterns assuming a forecast horizon ($H = 4$):

#### Small `step_size` ($S = 1 < H$) — Overlapping Windows
The calibration origin advances by 1 step at a time. This creates significant overlap between consecutive forecast windows, maximizing sample size ($n$) for short historical series.

```plaintext
Time Axis:      | t1 | t2 | t3 | t4 | t5 | t6 | t7 | t8 | t9 | t10|
------------------------------------------------------------------
Window 1:       [=== Initial Train ===]  [--- H=4 (t5 to t8) ---]
Window 2:       [==== Train + 1 ====]    [--- H=4 (t6 to t9) ---]   (Shifted S=1)
Window 3:       [===== Train + 2 =====]    [--- H=4 (t7 to t10) --] (Shifted S=1)
```

#### Default `step_size` ($S = H = 4$) — Disjoint Windows
The calibration origin shifts by the full forecast horizon ($S = H$). Each window starts exactly where the previous forecast ended, eliminating overlap and ensuring independence among calibration residuals.

```
Time Axis:      | t1 | t2 | t3 | t4 | t5 | t6 | t7 | t8 | t9 | t10| t11| t12|
----------------------------------------------------------------------------
Window 1:       [=== Initial Train ===]  [--- H=4 (t5 to t8) ---]
Window 2:       [======= Expanded Train =======] [--- H=4 (t9 to t12) --] (Shifted S=4)
```

#### Large step_size ($S = 6 > H$) — Windows with Gaps
The stride between windows exceeds the forecast horizon ($S > H$). This introduces temporal gaps between evaluation windows, mimicking real-world systems with infrequent retraining schedules.
```plaintext
Time Axis:      | t1 | t2 | t3 | t4 | t5 | t6 | t7 | t8 | t9 | t10| t11| t12| t13| t14|
----------------------------------------------------------------------------------------
Window 1:       [=== Initial Train ===]  [--- H=4 (t5 to t8) ---]
               |                      |                          |
               |<- Evaluated Train -->| <-- Gap (t9, t10) -----> | (Shifted S=6)
               |                      |                          v
Window 2:       [============ Expanded Train ============] [--- H=4 (t11 to t14) --]
```


## Classes

### Regression

Import these classes from `tinyconformal.regressor`:

- `ConformalizedRegressor`: conformalizes a fitted point regressor and produces
  prediction intervals. It supports split, OOB, and precomputed OOF-score
  calibration.
- `ConformalizedQuantileRegressor`: implements conformalized quantile regression
  (CQR) for learners that produce lower and upper quantile predictions.
- `CrossValidationCalibration`: generates OOF ICP/CQR scores and cross-fitted
  CPS location residuals, scales, and standardized residuals.

### Classification

Import these classes from `tinyconformal.classifier`:

- `BinaryMarginalConformalClassifier`: constructs binary prediction sets with
  marginal coverage.
- `BinaryClassConditionalConformalClassifier`: constructs binary prediction sets
  with coverage calibrated separately for each class.

Both classifiers support split calibration with `fit(X, y)`, OOB calibration
with `fit(y=y_train, oob=True)`, and precomputed OOF calibration with
`fit_from_probabilities(probabilities, y)`.

### Distribution

Import these classes from `tinyconformal.distribution`:

- `ContinuousCrossConformalPredictiveSystem`: cross-fits location and scale
  regressors and produces continuous predictive distributions.
- `DiscreteCrossConformalPredictiveSystem`: produces conformal predictive
  distributions for ordered integer or count targets.

### Time Series Distribution

Import these classes from `tinyconformal.series`:

- `MultiStepConformalTimeSeriesRegressor `: produces MSCP prediction bands
  from horizon-specific signed residuals.
- `ConformalizedQuantileTimeSeriesRegressor`: produces TSCQR prediction bands from
  horizon-specific conformalized quantile scores.
- `ContinuousTimeSeriesConformalPredictiveSystem`: produces a complete
  continuous predictive distribution for every series and forecast horizon.
- `DiscreteTimeSeriesConformalPredictiveSystem`: produces complete
  predictive distributions for ordered integer or count time-series targets and
  supports PMF evaluation.
The time-series CPS classes use rolling-origin calibration with
`fit(df, step_size=...)`. TSCPS exposes `predict_distribution` for multi-step
forecasts; the returned object provides quantiles, intervals, and CDFs.

## License

This project is licensed under the MIT License.
