Metadata-Version: 2.4
Name: sensebench
Version: 0.1.0
Summary: Glite SenseBench benchmark and leaderboard tooling for English word sense disambiguation evaluation
Project-URL: Homepage, https://github.com/GliteTech/sensebench
Project-URL: Repository, https://github.com/GliteTech/sensebench
Project-URL: Issues, https://github.com/GliteTech/sensebench/issues
Project-URL: Company, https://glite.ai
Author-email: Glite Tech Ltd <info@glite.ai>
Maintainer-email: Vassili Philippov <vassili@glite.ai>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,evaluation,leaderboard,lexen,llm,nlp,word-sense-disambiguation,wsd
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.12
Requires-Dist: jinja2>=3.1
Requires-Dist: litellm>=1.0
Requires-Dist: nltk>=3.9
Requires-Dist: numpy>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: tqdm>=4.66
Description-Content-Type: text/markdown

# SenseBench

SenseBench is benchmark and leaderboard tooling for evaluating English word sense disambiguation
(WSD) on lexEN and related datasets. A model is shown a target word in context plus its candidate
WordNet senses and must answer with the index of the correct sense. Prompts are immutable,
registered definitions; runs produce fully auditable artifacts that anyone can re-verify down to
the raw API responses.

## Install

Requires Python 3.12+.

```bash
uv tool install sensebench   # or: pipx install sensebench / pip install sensebench
```

For development, clone this repository and run `uv sync`.

## Configure API keys

SenseBench calls models through [LiteLLM](https://docs.litellm.ai/), so any supported provider
works. Export the provider key, or put it in a `.env` file in your working directory (see
`.env.example`):

```bash
export OPENAI_API_KEY=...
```

## Run a benchmark

```bash
sensebench run --model gpt-5.5 --prompt p001
```

On first use this downloads the NLTK WordNet corpus and the registered dataset release
(`lexen-v0.1.0`, cached under `~/.cache/sensebench/`, integrity-checked against a pinned SHA-256
hash). The run preflights the model with a single call, evaluates every item, writes
`runs/<run-id>/`, and prints an accuracy summary. Useful flags:

* `--limit 25` for a cheap smoke run (not leaderboard-eligible)
* `--run-id my-run` to name the run yourself (otherwise generated from model, prompt, and dataset)
* `--temperature`, `--max-tokens`, `--seed`, `--reasoning-effort` for sampling control
* `--hosting-kind self_hosted --endpoint-base-url http://...` for self-hosted models

Each run directory contains:

* `run.json` — run metadata, policy, and totals
* `predictions.jsonl` — one record per item with candidates, votes, and correctness
* `calls.jsonl.gz` — every raw API request and response

## Verify a run

Verification replays the full chain — prompt rendering, answer extraction, vote decisions, and
correctness against gold — from the stored artifacts:

```bash
sensebench verify runs/<run-id> --dataset lexen-v0.1.0 --prompt p001
```

## Prompts

Registered prompts are immutable JSON definitions under `src/sensebench/prompts/registered/`. Any
benchmark-relevant change requires a new prompt ID. See `docs/prompts.md`.

## Leaderboard

`sensebench leaderboard` aggregates verified run directories from `results/` into
`leaderboard.json`. Every run is re-verified before inclusion, and accuracy is recomputed from the
predictions rather than trusted from metadata.

## Website

The public leaderboard site is generated as static GitHub Pages output:

```bash
sensebench site build --results-dir results --output-dir _site --strict
```

The generated site includes an interactive leaderboard, Pareto charts, static run-detail pages,
dataset and prompt reference pages, submission instructions, a sitemap, and static JSON under
`_site/data/`. Pull requests that add public results must place complete run artifacts under
`results/<run-id>/`; CI verifies all submitted results and builds a preview artifact. Merges to
`main` deploy the rebuilt site automatically.

## Development

```bash
uv sync
uv run pytest
uv run ruff check src tests tools
uv run mypy src
uv run python tools/verify_prompt.py --all
uv run sensebench site build --results-dir results --output-dir _site --strict
```

## License

Apache-2.0.
