Metadata-Version: 2.4
Name: judge-bench
Version: 0.1.2
Summary: Synthetic bias and calibration probes for testing LLM-as-judge reliability.
Author: AuraOne
License-Expression: MIT
Project-URL: Homepage, https://github.com/auraoneai/judge-bench
Project-URL: Documentation, https://github.com/auraoneai/judge-bench#readme
Project-URL: Repository, https://github.com/auraoneai/judge-bench.git
Project-URL: Issues, https://github.com/auraoneai/judge-bench/issues
Project-URL: Changelog, https://github.com/auraoneai/judge-bench/blob/main/CHANGELOG.md
Keywords: llm-as-judge,judge-bias,model-evaluation,calibration,evals,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# judge-bench

Probe an LLM judge for position, verbosity, self-preference, paraphrase, anchoring, and calibration sensitivity.

`judge-bench` is for evaluation engineers choosing or monitoring an LLM-as-judge configuration. Unlike a model leaderboard, it runs controlled synthetic response pairs through a selected backend, records probe-level evidence, caches repeated calls, and produces reviewable report and plot artifacts.

## Inspectable Output

A run writes:

- `<name>.json`: backend, model, synthetic disclosure, cache hits/misses, and probe results.
- `<name>.md`: a concise diagnostic report.
- `<name>.plots.json`: plot-ready reliability and bias points.
- `<name>.svg`: deterministic plots.
- `<name>.png`: only when `matplotlib` is installed.

The built-in dry-run cost estimate uses static per-call constants for planning. It does not query provider pricing and should not be treated as a quote.

## Runtime Boundary

The default `local-judge` model is a deterministic lexical heuristic and makes no network request. `ollama:`, `vllm:`, and hosted Hugging Face modes call the configured local HTTP endpoint; `transformers:` loads a local Python model pipeline. The `openai`, `anthropic`, and `google` backends send the synthetic prompt/response pairs to those providers and require their standard API-key environment variables. Paid-backend runs require `--confirm-cost`.

Cache entries contain model inputs and outputs as local JSON files. Choose `--cache-dir` accordingly when prompts or rationales are sensitive.

## Install

```bash
python -m pip install judge-bench==0.1.2
```

For development from a clone:

```bash
python -m pip install -e .
```

## Quickstart

Run a fully offline diagnostic:

```bash
judge-bench run \
  --backend local \
  --model local-judge \
  --probes position_bias,calibration \
  --pairs 3 \
  --cache-dir .judge-bench-cache \
  --output judge-report.json
```

## Backends

- Hosted: `openai`, `anthropic`, `google`.
- Local HTTP: `ollama:<model>`, `vllm:<model>`, `hf:<model>`.
- Local Python: `transformers:<model>` with `transformers` installed.
- Offline smoke test: `local-judge`.

See [`docs/what-each-probe-measures.md`](docs/what-each-probe-measures.md), [`docs/interpreting-position-bias.md`](docs/interpreting-position-bias.md), and [`docs/methodology.md`](docs/methodology.md).

## Release Status

Registry status verified July 13, 2026: version `0.1.2` is published on PyPI and tagged `v0.1.2` in the public repository. The project is alpha software. No model-quality, superiority, or adoption claim is made.

## Limits

All bundled response pairs are synthetic. Results characterize the selected prompts, backend, model, and probe set; they do not establish general model quality or production safety.

## Next Action

Run the offline `local-judge` quickstart, inspect the probe-level Markdown and SVG evidence, then repeat with the intended production judge only after confirming the cache, data-sharing, network, and cost boundary.
