Metadata-Version: 2.4
Name: vlm-eval
Version: 0.1.0
Summary: Evaluation harness for Vision-Language Models on non-standard axes (calibration under distribution shift, and beyond)
Author: Suz99
License: MIT
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: numpy>=1.26
Requires-Dist: pillow>=10.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: clip
Requires-Dist: open-clip-torch>=2.24; extra == 'clip'
Requires-Dist: torch<2.12,>=2.4; extra == 'clip'
Description-Content-Type: text/markdown

# vlm-eval

A lightweight evaluation harness for Vision-Language Models (VLMs) on axes
that mainstream benchmarks don't cover. First task module: **calibration
under distribution shift** — how well a model's confidence tracks its
actual correctness as images are blurred or corrupted with noise.

## Quickstart

```bash
uv sync --all-extras --dev
uv run vlm-eval run --task calibration_shift --model mock
```

This runs the `calibration_shift` task against the deterministic
`MockVLMWrapper` over the tiny fixture dataset in `tests/fixtures/`, and
writes `report/leaderboard.md` + `report/leaderboard.json`.

Add `--shift blur` or `--shift noise` to evaluate under a corruption.

## Design

- `vlm_eval/models/` — thin wrappers around real VLM backends
  (`transformers`, `open_clip`), all implementing one `VLMWrapper`
  interface so tasks and scorers are model-agnostic.
- `vlm_eval/tasks/` — a task pairs a dataset with a prompt template and a
  scorer.
- `vlm_eval/scorers/` — accuracy, Expected Calibration Error (ECE), Brier
  score.
- `vlm_eval/data/` — dataset loading and distribution-shift transforms.
- `vlm_eval/report/` — markdown/JSON leaderboard output.

CI runs entirely offline against `MockVLMWrapper` and synthetic fixture
images — no real model downloads or GPU required. Running real VLMs
against real datasets is a manual/local workflow (see `models/clip_wrapper.py`).

## First real result: CLIP

`ClipWrapper` (`vlm_eval/models/clip_wrapper.py`) wraps `open_clip`'s
`ViT-B-32-quickgelu` / `openai` checkpoint and does standard zero-shot
classification — cosine similarity between the image embedding and a
`"a photo of a {label}"` text embedding per candidate label, softmaxed into
a probability distribution. It ignores the free-form `prompt` argument
(built for instruction-following models); CLIP compares an image against
labels directly.

```bash
uv sync --extra clip --dev
uv run vlm-eval run --task calibration_shift --model clip
uv run vlm-eval run --task calibration_shift --model clip --shift noise
```

On the tiny synthetic fixture set (solid dark/light gray squares):

| model | shift  | accuracy | ece    | brier_score |
| ----- | ------ | -------- | ------ | ----------- |
| clip  | none   | 0.5000   | 0.4188 | 0.7756      |
| clip  | blur   | 0.5000   | 0.4188 | 0.7756      |
| clip  | noise  | 1.0000   | 0.3365 | 0.2786      |

Blur is a no-op on uniform-color squares, as expected. CLIP essentially
guesses on "dark"/"light" gray squares — they're not natural photos, so
`"a photo of a dark"` isn't a meaningful query — which is itself a useful
data point: the harness surfaces a real, non-trivial calibration gap
(50% accuracy, but a non-zero ECE means its confidence still doesn't
track that chance-level correctness). A richer, more photographic
fixture set would give a more representative result; this one mainly
demonstrates the harness computing real metrics end-to-end against a
real model, not a benchmark claim about CLIP itself.

## Development

```bash
uv sync --all-extras --dev
uv run pytest tests/unit tests/integration
uv run ruff check .
uv run black --check .
uv run mypy vlm_eval/
```
