Metadata-Version: 2.5
Name: nonconform
Version: 2.0.1
Summary: Conformal Anomaly Detection
Project-URL: Homepage, https://github.com/OliverHennhoefer/nonconform
Project-URL: Documentation, https://oliverhennhoefer.github.io/nonconform/
Project-URL: Bugs, https://github.com/OliverHennhoefer/nonconform/issues
Author-email: Oliver Hennhoefer <oliver.hennhoefer@mail.de>
Maintainer-email: Oliver Hennhoefer <oliver.hennhoefer@mail.de>
License: BSD 3-Clause License
        
        Copyright (c) 2024, Oliver Hennhöfer
        
        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:
        
        1. Redistributions of source code must retain the above copyright notice, this
           list of conditions and the following disclaimer.
        
        2. Redistributions in binary form must reproduce the above copyright notice,
           this list of conditions and the following disclaimer in the documentation
           and/or other materials provided with the distribution.
        
        3. Neither the name of the copyright holder nor the names of its
           contributors may be used to endorse or promote products derived from
           this software without specific prior written permission.
        
        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
        AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
        IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
        DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
        FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
        DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
        SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
        CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
        OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
License-File: LICENSE
Keywords: anomaly detection,conformal anomaly detection,conformal inference,false discovery rate,uncertainty quantification
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: numpy>=2.2.0
Requires-Dist: pandas>=2.2.2
Requires-Dist: scikit-learn>=1.5.2
Requires-Dist: scipy>=1.13.0
Requires-Dist: tqdm>=4.66.2
Provides-Extra: all
Requires-Dist: kdepy>=1.1.12; extra == 'all'
Requires-Dist: oddball>=1.3.0; extra == 'all'
Requires-Dist: online-fdr>=1.0.0; extra == 'all'
Requires-Dist: optuna>=4.5.0; extra == 'all'
Requires-Dist: pyarrow>=16.1.0; extra == 'all'
Requires-Dist: pyod<4,>=2.0.7; extra == 'all'
Provides-Extra: data
Requires-Dist: oddball>=1.3.0; extra == 'data'
Requires-Dist: pyarrow>=16.1.0; extra == 'data'
Provides-Extra: fdr
Requires-Dist: online-fdr>=1.0.0; extra == 'fdr'
Provides-Extra: probabilistic
Requires-Dist: kdepy>=1.1.12; extra == 'probabilistic'
Requires-Dist: optuna>=4.5.0; extra == 'probabilistic'
Provides-Extra: pyod
Requires-Dist: pyod<4,>=2.0.7; extra == 'pyod'
Description-Content-Type: text/markdown

<h1 align="center">
  <a href="https://oliverhennhoefer.github.io/nonconform/">
    <img src="https://raw.githubusercontent.com/OliverHennhoefer/nonconform/main/docs/source/assets/banner.png" alt="nonconform" width="900">
  </a>
</h1>

<p align="center">
  <strong>Calibrate scores. Control discoveries. Monitor change.</strong>
</p>

<p align="center">
  <a href="https://pypi.org/project/nonconform/"><img src="https://img.shields.io/pypi/v/nonconform.svg" alt="PyPI version"></a>
  <a href="https://pypi.org/project/nonconform/"><img src="https://img.shields.io/pypi/pyversions/nonconform.svg" alt="Supported Python versions"></a>
  <a href="https://github.com/OliverHennhoefer/nonconform/actions/workflows/codecov.yml"><img src="https://img.shields.io/github/actions/workflow/status/OliverHennhoefer/nonconform/codecov.yml?branch=main&label=Tests" alt="Tests"></a>
  <a href="https://codecov.io/gh/OliverHennhoefer/nonconform"><img src="https://codecov.io/gh/OliverHennhoefer/nonconform/branch/main/graph/badge.svg?token=Z78HU3I26P" alt="Code coverage"></a>
  <a href="https://oliverhennhoefer.github.io/nonconform/"><img src="https://img.shields.io/github/actions/workflow/status/OliverHennhoefer/nonconform/docs.yml?branch=main&label=Documentation" alt="Documentation"></a>
  <a href="https://github.com/OliverHennhoefer/nonconform/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/nonconform.svg" alt="License"></a>
</p>

<p align="center">
  <a href="https://oliverhennhoefer.github.io/nonconform/">Documentation</a> ·
  <a href="https://oliverhennhoefer.github.io/nonconform/quickstart/">Batch workflow</a> ·
  <a href="https://oliverhennhoefer.github.io/nonconform/user_guide/exchangeability_martingales/">Sequential workflow</a> ·
  <a href="https://oliverhennhoefer.github.io/nonconform/api/">API reference</a> ·
  <a href="https://arxiv.org/abs/2605.13642">Paper</a>
</p>

`nonconform` calibrates anomaly scores for discovery, threshold
certification, and sequential change monitoring. Wrap a
supported scikit-learn estimator, a [PyOD](https://pyod.readthedocs.io/) model,
or a custom detector:

- **Discovery:** Use calibrated p-values directly or call `select(...)` to apply
  false discovery rate (FDR) control.
- **Score thresholds:** Build a simultaneous false-alarm/FPR certificate and
  choose a score threshold for future inliers.
- **Stream:** Use conformal martingales to accumulate evidence against
  exchangeability and trigger configured alarms.

## Why nonconform?

- **Calibrate anomaly scores** into conformal p-values using reference data.
- **Certify raw-score thresholds** with simultaneous FPR certificates and
  target-driven threshold inversion.
- **Control discoveries** by accounting for multiple tests within a
  fixed family.
- **Monitor streams for change** with conformal martingales, anytime evidence
  against exchangeability, and configurable alarms.
- **Keep your detector** through support for PyOD, recognized scikit-learn
  estimators, and protocol-compliant custom models.

<p align="center">
  <strong>Works with</strong>
  &nbsp;&nbsp;&nbsp;
  <a href="https://scikit-learn.org/"><img src="https://raw.githubusercontent.com/OliverHennhoefer/nonconform/main/docs/img/integrations/scikit-learn.svg?v=dc048ee" alt="scikit-learn" height="44" align="middle"></a>
  &nbsp;&nbsp;&nbsp;&nbsp;
  <a href="https://pyod.readthedocs.io/"><img src="https://raw.githubusercontent.com/OliverHennhoefer/nonconform/main/docs/img/integrations/pyod.svg?v=1" alt="PyOD" height="44" align="middle"></a>
  &nbsp;&nbsp;&nbsp;&nbsp;
  <a href="https://oliverhennhoefer.github.io/nonconform/user_guide/detector_compatibility/"><img src="https://raw.githubusercontent.com/OliverHennhoefer/nonconform/main/docs/img/integrations/custom.svg?v=1" alt="Custom AnomalyDetector protocol" height="44" align="middle"></a>
</p>

## Installation

`nonconform` requires Python 3.12 or newer. Both discovery control and
sequential monitoring are included in the core installation.

```bash
pip install nonconform
```

For the PyOD detector collection and benchmark datasets:

```bash
pip install "nonconform[pyod,data]"
```

<details>
<summary><strong>Optional extras</strong></summary>

| Extra | Adds |
| --- | --- |
| `pyod` | PyOD anomaly detectors |
| `data` | `oddball` benchmark datasets and PyArrow support |
| `fdr` | Streaming FDR procedures from `online-fdr` |
| `probabilistic` | KDE-based probabilistic estimation and tuning |
| `all` | Every optional feature |

</details>

## Quick start

### Batch discovery control

This core-only example demonstrates the batch lane. The detector is trained on
normal data, part of which is reserved automatically for conformal calibration.

```python
import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_train = rng.normal(size=(3_000, 2))
x_test = np.vstack([
    rng.normal(size=(200, 2)),
    rng.normal(loc=5.0, size=(20, 2)),
])

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_train)

discoveries = detector.select(x_test, alpha=0.05)
p_values = detector.last_result.p_values

print(f"Selected {discoveries.sum()} of {len(x_test)} observations")
```

> [!NOTE]
> `discoveries` is a Boolean mask. Here, `alpha=0.05` is the target FDR level,
> not a per-observation score threshold. The underlying conformal p-values remain
> available through `last_result` for inspection or downstream analysis.

For threshold exploration in one fixed batch, use
`certificate = detector.fdp_bounds(x_test, confidence=0.95)` and query its
realized-FDP upper curve. `certificate.select(0.01)` applies the p-value cutoff
0.01; it does not request FDP at most 1%. The ordinary call prepares one
default envelope using fresh Monte Carlo sampling, and subsequent queries
reuse it. Coverage is joint over data and that sampling. Keep the scoring rule
and family fixed, and do not cherry-pick repeated constructions. The
[FDP guide](https://oliverhennhoefer.github.io/nonconform/user_guide/fdr_control/#post-hoc-simultaneous-fdp-bounds)
shows a complete example and the separate expert interface for method settings.

### Raw-score false-alarm control

If the operational requirement is “flag no more than 5% of future inliers,” use
the raw-score certificate instead of treating an FDR target as a score
threshold:

```python
certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)
print(f"Certified threshold: {certificate.threshold_for(0.05)}")
```

With probability at least 95% over the clean calibration draw, the certificate
bounds the population inlier false-alarm rate at every threshold, conditional
on the independently fitted score map. On that event, the chosen threshold has
a future-inlier false-alarm probability at most 5%. Calibration and future
inlier scores must be i.i.d. from the same distribution; exchangeability alone
is insufficient.

The certificate uses the deterministic finite-sample one-sided KS distribution.
With 1,000 calibration scores at 95% confidence, its finite-threshold band floor
is about 3.85%, allowing the 5% target here. Smaller targets can produce an
infinite threshold and an empty selection. Native certificates support
unweighted `Split` calibration independently of the p-value estimator. This
does not guarantee a realized false-alarm proportion in a finite batch, FDR,
FDP, recall, or sequential Ville control.

### Sequential change monitoring

A fitted `Split` detector can initialize the stream lane without refitting its
scoring model. The example is self-contained so it can be copied independently
of the batch example.

<details>
<summary><strong>Show sequential monitoring example</strong></summary>

```python
import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor

rng = np.random.default_rng(42)
x_train = rng.normal(size=(1_000, 2))

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
).fit(x_train)

alpha = 0.05
monitor = ExchangeabilityMonitor.from_split_detector(
    detector,
    martingale=SimpleJumperMartingale(
        alarm_config=AlarmConfig(restarted_ville_threshold=1 / alpha)
    ),
    seed=42,
)

# Stable observations followed by a distribution shift
x_stream = np.vstack([
    rng.normal(size=(50, 2)),
    rng.normal(loc=3.0, size=(50, 2)),
])

for x_t in x_stream:
    state = monitor.update(x_t)
    if "restarted_ville" in state.triggered_alarms:
        print(f"Change alarm at step {state.evidence_step}")
        break
```

</details>

Under the sequential validity assumptions, the restarted Ville alarm at
`1 / alpha` controls the probability of ever crossing on one stream. It does
not control FDR across multiple streams. See the
[sequential monitoring guide](https://oliverhennhoefer.github.io/nonconform/user_guide/exchangeability_martingales/)
for the full guarantee scope and other alarm statistics.

## Choose a workflow

| Goal | Start with |
| --- | --- |
| Calibrate and select anomalies in a batch | [`Split` and `select(...)`](https://oliverhennhoefer.github.io/nonconform/quickstart/) |
| Set a per-inlier raw-score false-alarm target | [`fpr_bounds(...)` and FPR certificates](https://oliverhennhoefer.github.io/nonconform/user_guide/fdr_control/#raw-score-false-alarm-control) |
| Aggregate evidence across random splits | [`DerandomizedSplits` and e-BH](https://oliverhennhoefer.github.io/nonconform/examples/derandomized_e_values/) |
| Monitor a stream for change | [Exchangeability martingales](https://oliverhennhoefer.github.io/nonconform/user_guide/exchangeability_martingales/) |
| Reuse more data for fitting and calibration | [`CrossValidation` or `JackknifeBootstrap`](https://oliverhennhoefer.github.io/nonconform/user_guide/conformalization_strategies/) |
| Account for covariate shift | [Weighted conformal inference](https://oliverhennhoefer.github.io/nonconform/user_guide/weighted_conformal/) |
| Calibrate groups with different normal behavior | [Group-conditional (Mondrian) p-values](https://oliverhennhoefer.github.io/nonconform/user_guide/conformal_inference/#group-conditional-mondrian-p-values) |
| Certify realized FDP across p-value thresholds | [`fdp_bounds(...)` and FDP certificates](https://oliverhennhoefer.github.io/nonconform/user_guide/fdr_control/#post-hoc-simultaneous-fdp-bounds) |
| Bring a custom or third-party detector | [Detector compatibility](https://oliverhennhoefer.github.io/nonconform/user_guide/detector_compatibility/) |

## Statistical scope

> [!IMPORTANT]
> **Guarantees are assumption-dependent.** Standard conformal workflows require
> calibration data and null test cases to be exchangeable. FDR claims additionally
> require valid p-values or the applicable aggregate null-evidence condition,
> together with the assumptions of the selected multiple-testing procedure.
> FPR certificates require clean, i.i.d. calibration and future inlier scores
> from the same distribution, conditional on an independently fitted score map.
> Weighted workflows require a plausible covariate-shift model, support
> overlap, and reliable weights. Sequential martingales require valid sequential
> conformal p-values; Ville thresholds provide false-alarm control for one valid
> stream, while CUSUM and Shiryaev-Roberts thresholds are change-evidence triggers
> that require separate calibration.

`nonconform` calibrates detector scores; it cannot make an unsuitable detector
or mismatched calibration set valid. Spatial or temporal dependence must be
handled explicitly before applying standard exchangeability-based claims. See
the guides to [FDR control](https://oliverhennhoefer.github.io/nonconform/user_guide/fdr_control/)
and [sequential monitoring](https://oliverhennhoefer.github.io/nonconform/user_guide/exchangeability_martingales/)
before relying on error-control statements in a new application.

## Citation

If you use `nonconform` in academic work, please cite the
[accompanying paper](https://arxiv.org/abs/2605.13642):

```bibtex
@inproceedings{Hennhoefer2026,
author = {Hennhöfer, Oliver and Kirsch, Maximilian and Preisach, Christine},
booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
pages = {613--632},
publisher = {PMLR},
title = {{Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform}},
url = {https://proceedings.mlr.press/v329/hennhofer26a.html},
year = {2026}
}
```

## Project

Read the [documentation](https://oliverhennhoefer.github.io/nonconform/), browse
the [changelog](https://github.com/OliverHennhoefer/nonconform/blob/main/CHANGELOG.md),
or report a problem in the [issue tracker](https://github.com/OliverHennhoefer/nonconform/issues).
Contributions are welcome; start with the
[contributing guide](https://oliverhennhoefer.github.io/nonconform/contributing/).
`nonconform` is distributed under the
[BSD 3-Clause License](https://github.com/OliverHennhoefer/nonconform/blob/main/LICENSE).

---

<p align="center">
  <a href="https://www.dlr.de/">
    <img src="https://www.dlr.de/de/pt-lf/aktuelles/pressematerial/logos/bmwk/vorschaubild_bmwk_logo-mit-foerderzusatz_en/@@images/image-600-ea91cd9090327104991124b30fe1de7d.png" alt="Funding acknowledgement" width="250">
  </a>
</p>
