Metadata-Version: 2.5
Name: judgekeeper
Version: 0.1.2
Summary: Your AI judge grades your app; judgekeeper grades the judge against human labels.
Project-URL: Homepage, https://www.judgekeeper.com
Project-URL: Source, https://github.com/judgekeeper/judgekeeper
Project-URL: Changelog, https://github.com/judgekeeper/judgekeeper/blob/main/CHANGELOG.md
Author: Sathvik Thota
License-Expression: MIT
License-File: LICENSE
Keywords: calibration,custom-metric,drift,evaluation,llm,llm-as-judge,rubric
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: build>=1; extra == 'dev'
Requires-Dist: hatchling>=1.26; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: inspect
Requires-Dist: inspect-ai>=0.3; extra == 'inspect'
Provides-Extra: mlflow
Requires-Dist: mlflow>=3.1; extra == 'mlflow'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# judgekeeper

**Your AI judge grades your app. judgekeeper grades the judge.**

It compares the judge's verdicts with answers real people gave, and tells you if the judge can be trusted.

## Try it

```
pip install judgekeeper
judgekeeper demo
```

It needs no key and no data. The demo checks a recorded judge on answers that ship with the package. Without installing: `uvx judgekeeper demo`. On a Mac the command is `pip3`. If the install is refused, see [Install](https://www.judgekeeper.com/start.html#install).

## Use it on your own judge

```
judgekeeper check results.csv --judge verdict --human label
```

The table has one row per answer: what the judge said and what a person said.

## What you get

```
judgekeeper demo: a recorded judge on 100 synthetic single-output items, 3 runs.
Of the answers people passed, the judge passed 97%. Of the answers people failed, the judge failed 86%. It is usable with care.
TPR 0.97 (95% CI 0.88 to 0.99), TNR 0.86 (95% CI 0.73 to 0.94), kappa 0.84
Usable with care: TPR 0.97, TNR 0.86, kappa 0.84 against human labels. Read the flags below before relying on it.
  - TNR is 0.86 (between 0.80 and 0.90): usable with care.
open judgekeeper-demo/report.html (data in judgekeeper-demo/report.json)
```

The second line is the answer in plain words. The next two lines give the same result as numbers; [How it works](https://www.judgekeeper.com/learn.html#numbers) says how to read them. The indented line is a flag: what to look at before you rely on this judge. The last line is a report page you can open.

A report from a real judge on a public dataset: [LLMBar judged by Claude Haiku 4.5](https://www.judgekeeper.com/examples/llmbar-haiku/report.html). Three worked examples with two judges each: [See it work](https://www.judgekeeper.com/examples.html).

## When you need more

| You need | Command | Read |
|---|---|---|
| Human labels, because you have none yet | `judgekeeper label items.jsonl` | [Labels](https://www.judgekeeper.com/start.html#step-label) |
| Your own rule, or judgekeeper to run the judge | `judgekeeper init` | [Your own metric](https://www.judgekeeper.com/own-metric.html) |
| A warning when the judge changes | `judgekeeper gate report.json` | [Keep checking](https://www.judgekeeper.com/learn.html#keeps) |
| To use results from an eval framework, pytest or CI | `judgekeeper import promptfoo results.json` | [Your tools](https://www.judgekeeper.com/learn.html#works) |
| A coding agent to set it up for you | none: the agent reads one file, `skills/judgekeeper/SKILL.md` | [With a coding assistant](https://www.judgekeeper.com/assistant.html) |

## Status

Alpha (0.1.2): commands and report fields may still change. [Use it on your app](https://www.judgekeeper.com/start.html) has the four steps, [How it works](https://www.judgekeeper.com/learn.html) has the background, the [reference](https://www.judgekeeper.com/reference.html) has every command and flag, and the [changelog](https://github.com/judgekeeper/judgekeeper/blob/main/CHANGELOG.md) says what changed.

## License

MIT.
