Metadata-Version: 2.4
Name: rankcal
Version: 0.2.0
Summary: Calibration and uncertainty quantification for ranking systems
Author: John Hodge
License: MIT
Project-URL: Homepage, https://github.com/jman4162/rankcal
Project-URL: Repository, https://github.com/jman4162/rankcal
Project-URL: Documentation, https://jman4162.github.io/rankcal/
Keywords: machine-learning,calibration,ranking,uncertainty-quantification,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: matplotlib>=3.5.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Dynamic: license-file

# rankcal

[![PyPI version](https://badge.fury.io/py/rankcal.svg)](https://badge.fury.io/py/rankcal)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://jman4162.github.io/rankcal/)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jman4162/rankcal/blob/main/examples/tutorial.ipynb)

Calibration and uncertainty quantification for ranking systems. PyTorch-first.

## Why rankcal?

Existing calibration libraries treat calibration as a classification problem. But ranking decisions happen at the **top-k**, and miscalibration there is what actually breaks business outcomes.

rankcal provides:
- **Ranking-aware calibration metrics** - ECE@k, top-k reliability diagrams
- **Monotonic calibrators** - Temperature scaling, isotonic regression, splines, neural networks
- **Decision analysis** - Risk-coverage curves, utility optimization

## Installation

```bash
pip install rankcal
```

For development:
```bash
pip install -e ".[dev]"
```

## Quick Start

```python
import torch
from rankcal import TemperatureScaling, ece_at_k, reliability_diagram

# Your ranking scores and binary relevance labels
scores = torch.tensor([0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1])
relevance = torch.tensor([1, 1, 0, 1, 0, 0, 1, 0, 0])

# Fit a calibrator
calibrator = TemperatureScaling()
calibrator.fit(scores, relevance)

# Calibrate scores
calibrated = calibrator(scores)

# Evaluate calibration at top-k
ece = ece_at_k(calibrated, relevance, k=5)
print(f"ECE@5: {ece:.4f}")

# Visualize calibration
fig = reliability_diagram(calibrated, relevance, k=5)
fig.savefig("reliability.png")
```

## Calibrators

| Calibrator | Differentiable | Parametric | Description |
|------------|----------------|------------|-------------|
| `TemperatureScaling` | ✓ | ✓ | Single learned temperature parameter |
| `IsotonicCalibrator` | ✗ | ✗ | Non-parametric, piecewise constant |
| `PiecewiseLinearCalibrator` | ✓ | ✓ | Monotonic piecewise linear interpolation |
| `MonotonicNNCalibrator` | ✓ | ✓ | Neural network with monotonicity constraints |

## GPU Support

All calibrators are PyTorch `nn.Module` subclasses and support GPU acceleration:

```python
import torch
from rankcal import TemperatureScaling

# Move calibrator to GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
calibrator = TemperatureScaling().to(device)

# Fit with data on GPU
scores = scores.to(device)
labels = labels.to(device)
calibrator.fit(scores, labels)

# Inference on GPU
test_scores = test_scores.to(device)
calibrated = calibrator(test_scores)
```

Run GPU tests with:
```bash
pytest tests/test_gpu.py --device cuda  # or --device mps on Mac
```

## Metrics

- `ece(scores, labels)` - Expected Calibration Error
- `ece_at_k(scores, labels, k)` - ECE computed only on top-k items
- `reliability_diagram(scores, labels, k=None)` - Visualization of calibration

## Citation

If you use rankcal in academic work, please cite:

```bibtex
@software{hodge2026rankcal,
  author = {Hodge, John},
  title = {rankcal: Calibration for Ranking Systems},
  year = {2026},
  url = {https://github.com/jman4162/rankcal},
  version = {0.2.0}
}
```

## License

MIT
