Metadata-Version: 2.4
Name: visaebench
Version: 0.1.0
Summary: A benchmark for evaluating Sparse Autoencoders (SAEs) on Vision Transformers
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/vraj130/visaebench
Project-URL: Repository, https://github.com/vraj130/visaebench
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.6
Requires-Dist: torchvision
Requires-Dist: transformers>=4.36
Requires-Dist: timm>=1.0
Requires-Dist: pillow
Requires-Dist: numpy
Requires-Dist: scikit-learn>=1.3
Requires-Dist: huggingface-hub>=0.20
Requires-Dist: datasets>=2.14
Requires-Dist: tqdm
Requires-Dist: pyyaml
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# VISAEBench

[![Tests](https://github.com/vraj130/visaebench/actions/workflows/test.yml/badge.svg)](https://github.com/vraj130/visaebench/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/visaebench.svg)](https://pypi.org/project/visaebench/)

VISAEBench is a benchmark for evaluating Sparse Autoencoders (SAEs) trained on Vision Transformer (ViT) patch activations. It scores an SAE across four interpretability dimensions (reconstruction, concept detection, spatial coherence, disentanglement) using 7 metrics (M1 through M7). It supports cross-backbone evaluation across CLIP, DINOv2, SigLIP, MAE, and DeiT (all ViT-B/16 or ViT-B/14).

## Installation

```bash
# From PyPI (after v0.1.0 release)
pip install visaebench

# From source (development)
git clone https://github.com/vraj130/visaebench && cd visaebench
pip install -e .
```

## Quickstart

```python
import visaebench
from visaebench.hub import load_sae

sae = load_sae("visaebench/clip-vitb16-saes", subfolder="batchtopk_16x_k128")
results = visaebench.evaluate(
    sae=sae,
    backbone_name="clip_vitb16",
    imagenet_path="/path/to/imagenet/val",  # or None to stream from HF
)
for r in results:
    print(f"{r.name:30s} {r.value:.4f}")
```

## Available SAE checkpoints

One HuggingFace repo per backbone; each holds 12 configs as subfolders (`batchtopk_{8x,16x,32x}_k{64,128,192,256}`).

| Backbone | HuggingFace repo |
|---|---|
| CLIP ViT-B/16 | `visaebench/clip-vitb16-saes` |
| DINOv2 ViT-B/14 | `visaebench/dinov2-vitb14-saes` |
| SigLIP ViT-B/16 | `visaebench/siglip-vitb16-saes` |
| MAE ViT-B/16 | `visaebench/mae-vitb16-saes` |
| DeiT ViT-B/16 | `visaebench/deit-vitb16-saes` |

The detailed 12-config table (all `batchtopk_{8x,16x,32x}_k{64,128,192,256}` subfolders) lives in [`docs/quickstart.md`](docs/quickstart.md).

## Documentation

- [`docs/quickstart.md`](docs/quickstart.md): end-to-end quickstart and the full checkpoint config table.
- [`docs/custom_models.md`](docs/custom_models.md): evaluating your own SAE and custom backbones.
- [`docs/metrics.md`](docs/metrics.md): detailed description of the 7 metrics (M1 through M7).

## Metrics

| # | Registry key | Dimension | Description |
|---|---|---|---|
| M1 | `localization` | spatial_coherence | Per-feature spatial coherence on patch grid |
| M2 | `fvu` | reconstruction | Fraction of variance unexplained |
| M3 | `downstream_preservation` | reconstruction | Linear probe accuracy after reconstruction vs raw |
| M4 | `sparse_probing` | concept_detection | Sparse logistic probe AUC over class labels |
| M5 | `monosemanticity` | concept_detection | Cross-model feature consistency |
| M6 | `cross_domain` | concept_detection | Generalization to OOD datasets (EuroSAT, DTD) |
| M7 | `absorption` | disentanglement | WordNet-hierarchy concept absorption rate |

## Citation

```bibtex
@inproceedings{visaebench2026,
  title     = {ViSAEBench: Cross-Backbone Evaluation of Vision Sparse Autoencoders Reveals Backbone-Dominated Variance and Metric Dissociations},
  author    = {TODO: maintainer to fill in author list},
  booktitle = {ICML 2026 Mechanistic Interpretability Workshop},
  year      = {2026}
}
```

## License

Apache 2.0. See [LICENSE](LICENSE).
