Metadata-Version: 2.4
Name: juror
Version: 0.1.0
Summary: Zero-cost LLM evaluation toolkit: score model outputs against rubrics using a local LLM-as-judge, and track consistency and drift over time.
Project-URL: Homepage, https://github.com/chatbot-1/juror
Project-URL: Issues, https://github.com/chatbot-1/juror/issues
Author-email: Atul Singh <getupmaverick@gmail.com>
License: MIT
License-File: LICENSE
Keywords: drift,eval,evaluation,llm,llm-as-judge,observability,ollama,rubric
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: ollama>=0.3.0
Provides-Extra: dashboard
Requires-Dist: pandas>=2.0; extra == 'dashboard'
Requires-Dist: streamlit>=1.30; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# juror

**A zero-cost LLM evaluation toolkit.** Score model outputs against a rubric using a
local model as the judge ("LLM-as-judge"), and track how consistent a model is across
repeated runs and across different models — running entirely on
[Ollama](https://ollama.com) with **no external API cost**.

> Status: v0.1.0. Scoring engine, batch evaluation, drift/consistency metrics, latency
> tracking, SQLite storage, and a Streamlit dashboard are all built and tested — 35 tests
> passing, including a real end-to-end run against a local model.

---

## What it does

You give `juror`:

1. a **rubric** — a set of criteria like *correctness*, *clarity*, *follows instructions*, and
2. a **response** from some model,

and it uses a *second* local model to score that response against each criterion, with a
short reason for every score. Because it runs on Ollama, evaluating costs nothing and
needs no API key.

It also measures things you can only measure when the model runs on your own machine:
**run-to-run consistency**, **model-to-model drift**, and **real latency**.

---

## Quick start

```bash
# 1. Install Ollama (https://ollama.com), then pull a model:
ollama pull llama3.2

# 2. Install juror:
pip install juror        # or, from source:  pip install -e ".[dev]"
```

```python
from juror import Criterion, Rubric, score

rubric = Rubric(
    name="qa",
    criteria=[
        Criterion("correctness", "Is the answer factually correct?"),
        Criterion("clarity", "Is the answer clear and easy to understand?"),
    ],
)

result = score(
    prompt="What is the capital of France?",
    response="The capital of France is Paris.",
    rubric=rubric,
    model="llama3.1",
)

for s in result.scores:
    print(s.criterion, s.score, "--", s.reasoning)
print("overall:", result.overall())
```

See [`examples/quickstart.py`](examples/quickstart.py) for a runnable version.

---

## How the LLM-as-judge scoring works

`juror` builds a judge prompt that contains your rubric, the original prompt, and the
response to score, and asks the model to return JSON with a score and one-sentence reason
per criterion. Scores are parsed, clamped to each criterion's range, and returned as a
structured `JudgeResult`.

The judge is decoupled from the model behind a tiny `ModelClient` interface, which is what
makes it (a) unit-testable with a fake client and (b) swappable between models like
Llama 3.1 and Mistral.

### Known limitations (read this)

- **The judge can share blind spots with the model being judged.** If both models are wrong
  in the same way, the score can look confidently fine. LLM-as-judge is a useful, scalable
  signal — not ground truth.
- **Judges have biases** (e.g. rewarding longer or more confident answers). Part of this
  project is measuring how *consistent* the judge is, precisely because a single score
  can't be fully trusted.
- Small local models are noisier judges than large hosted ones. That trade-off — cost and
  privacy vs. raw judge quality — is the point, not a bug.

---

## Architecture

```
prompt + response ──► judge prompt ──► local model (Ollama) ──► JSON scores
                                                                    │
                                                        parse + clamp to rubric
                                                                    │
                                                              JudgeResult
                                                          (─► SQLite ─► dashboard)
```

---

## Sample results

All numbers below are **real output** from evaluating the 18 built-in starter prompts on
two local models — **llama3.2 (3B)** and **gemma2:2b** — on a 16 GB laptop (CPU), with each
model judging its own answers ("self-eval"). 84 grades in total. Reproduce with
[`examples/generate_demo_data.py`](examples/generate_demo_data.py).

### Model comparison — quality vs. speed

![Model comparison: quality vs. speed](docs/screenshots/dashboard-comparison.png)

| Model | Avg score / 5 | Median latency |
|---|---|---|
| **llama3.2 (3B)** | 3.68 | **14.4 s** |
| **gemma2:2b** | 4.53 | **16.6 s** |

The two are close on speed, but gemma scores its own answers noticeably higher.

> **Why _median_ latency, not mean?** The first grade on the freshly-pulled gemma model took
> **1148 s** — a one-time cost to load the model into memory. Mean latency would report a
> misleading ~43 s; median (16.6 s) reflects steady-state performance. You can only see this
> *because* the model runs on your machine — hosted APIs hide the infrastructure.

### A judge-bias caveat, made visible

Because these are *self-evals*, a higher score doesn't prove a better model — it can mean a
more **lenient judge**. gemma gave *itself* a perfect **5.00** on completeness, faithfulness,
*and* format; llama gave itself **3.00** on format and **3.11** on conciseness. LLM-as-judge
produces *relative signals*, not absolute truth — exactly the limitation flagged above.

### Run-to-run consistency

![Run-to-run consistency, one bar per model](docs/screenshots/dashboard-consistency.png)

Same prompt, 5 runs each at temperature 0.8 — std dev of the overall score (lower = steadier):

| Prompt | llama3.2 | gemma2:2b |
|---|---|---|
| Sentiment classification | **0.837** | 0.200 |
| Topic classification | 0.583 | **0.000** |
| "What causes the seasons?" | 0.490 | **0.000** |
| Summarization | 0.400 | 0.21–0.27 |
| "Capital of Japan?" | 0.267 | 0.133 |

llama wobbles most on **classification** — the borderline, subjective tasks. gemma looks
steadier, but largely because it's a lenient grader parked near the top of the scale (a
*ceiling effect*): **lower wobble isn't automatically "better."** Reading consistency and
calibration *together* is the whole point.

### Dashboard

![Dashboard: average score by criterion](docs/screenshots/dashboard-scores.png)

`juror-dashboard` opens this local Streamlit view — score-by-criterion, model comparison,
per-prompt consistency, score history, and a raw-data table — read live from your SQLite
results.

---

## Design decisions

- **Why LLM-as-judge instead of human labels?** Human labeling doesn't scale cheaply;
  LLM-as-judge is an increasingly standard pattern. The trade-off (shared blind spots) is
  documented above rather than hidden.
- **Why local models instead of a hosted API?** Zero cost, no rate limits, full
  reproducibility (you control the exact weights), and data privacy.
- **Why SQLite?** Zero setup, file-based, perfect for building up a history of eval runs
  you can chart over time.

---

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT — see [LICENSE](LICENSE).
