Metadata-Version: 2.4
Name: ecuas
Version: 0.1.0
Summary: Expected Cost for Uncertainty-Augmented Systems — metrics for evaluating calibration and selective prediction
Author-email: Erik Ernst <erik_ernst@hotmail.com.ar>
License-File: LICENSE
Requires-Python: ~=3.13.0
Requires-Dist: lightning>=2.6.4
Requires-Dist: numpy>=2.4.6
Requires-Dist: rich>=15.0.0
Requires-Dist: torch-uncertainty>=0.11.0
Requires-Dist: torch>=2.12.0
Requires-Dist: torchmetrics>=1.9.0
Requires-Dist: torchvision>=0.27.0
Description-Content-Type: text/markdown

# ECUAS: Expected Cost for Uncertainty-Augmented Systems

`ecuas` is a Python library containing robust calibration and classification evaluation metrics for Uncertainty-Augmented systems. It implements expected cost metrics, ECE, Brier Score, Cross Entropy, AUC, CCAS, LogLog, and selective prediction metrics under a unified `torchmetrics.Metric` interface.

## Installation

### Via `uv` (Recommended)
Add `ecuas` directly to your project:
```bash
uv add ecuas
```

### Via `pip`
You can install `ecuas` from PyPI:
```bash
pip install ecuas
```

## Features and Metrics

### Confidence/Selective Prediction Metrics
- **Expected Calibration Error (ECE)**: `ExpectedCalibrationError`
- **Confidence Error Rate**: `ConfidenceErrorRate`
- **Confidence AUC Score**: `ConfidenceAUCScore`
- **Confidence Brier Score**: `ConfidenceBrierScore`
- **Confidence Cross-Entropy**: `ConfidenceCrossEntropy`
- **Confidence ECUAS** (n-ECUAS): `ConfidenceECUAS`
- **Confidence Gamma-ECUAS**: `ConfidenceGammaECUAS`
- **Confidence AURC**: `ConfidenceAURC`
- **CCAS** (Confidence Cost for Selective Prediction): `CCAS`

### Classification Metrics
- **Classification Error Rate**: `ClassificationErrorRate`
- **Classification Cross-Entropy**: `ClassificationCrossEntropy`
- **Classification Brier Score**: `ClassificationBrierScore`
- **Classification AUC**: `ClassificationAUC`
- **Classification ECE**: `ClassificationECE`
- **Classification ECUAS**: `ClassificationECUAS`
- **Classification LogLog**: `ClassificationLogLog`
- **Classification Gamma-ECUAS**: `ClassificationGammaECUAS`
- **Classification AURC**: `ClassificationAURC`

## Usage Example

```python
import torch
from ecuas import ConfidenceECUAS, ExpectedCalibrationError

# Setup data
confidences = torch.tensor([0.9, 0.8, 0.4, 0.9])
correctness = torch.tensor([True, True, False, False])

# Expected Calibration Error
ece_metric = ExpectedCalibrationError(n_bins=10)
ece_metric.update(confidences, correctness)
ece_val = ece_metric.compute()
print(f"ECE: {ece_val.item():.4f}")

# Confidence n-ECUAS
ecuas_metric = ConfidenceECUAS(n=0)
ecuas_metric.update(confidences, correctness)
ecuas_val = ecuas_metric.compute()
print(f"ECUAS (n=0): {ecuas_val.item():.4f}")
```

## Running Tests

Execute the unit test suite:
```bash
uv run pytest
```

## License

This project is licensed under the Apache License 2.0. See the `LICENSE` file for details.
