Metadata-Version: 2.3
Name: twelve-angry-llms
Version: 0.2.0
Summary: Panel-based reliability annotation for preference data: measure inter-judge agreement (IJA) with a panel of LLM judges and export TRL-ready DPO data with reliability signals attached.
Keywords: llm-as-a-judge,preference-data,dpo,rlhf,data-curation,inter-judge-agreement
Author: meghdadFar
Author-email: meghdadFar <meghdad.farahmand@gmail.com>
License: MIT License
         
         Copyright (c) 2025 Meghdad Farahmand
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
         in the Software without restriction, including without limitation the rights
         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
         copies of the Software, and to permit persons to whom the Software is
         furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all
         copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
Requires-Dist: openai>=1.40.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: datasets>=2.19.0 ; extra == 'data'
Requires-Python: >=3.11
Project-URL: Repository, https://github.com/meghdadFar/twelve-angry-llms
Provides-Extra: data
Description-Content-Type: text/markdown

# Twelve Angry LLMs

**Panel-based reliability annotation for preference data.**

Modern preference datasets are built by asking a *single* strong judge to
score or rank candidate responses, then collapsing its verdict into
`(chosen, rejected)` pairs for DPO or reward-model training. That collapse
throws away any notion of how *contestable* each label was.

`twelve-angry-llms` recovers that signal: it runs a **panel of diverse LLM
judges** over each `(prompt, K responses)` datapoint and measures how much
the judges agree with each other — the **inter-judge agreement (IJA)**,
computed as average pairwise Kendall's τ-b over the judges' rankings. High
IJA means the label is trustworthy; low IJA means it is shaky and likely to
inject noise into training. The result exports straight into TRL's
`(prompt, chosen, rejected)` schema with the reliability signals attached,
so you can filter, down-weight, or soft-label your data and run your
existing training setup unchanged.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

## Installation

```bash
pip install twelve-angry-llms          # core: OpenRouter + any OpenAI-compatible provider
pip install "twelve-angry-llms[data]"  # + raw UltraFeedback / Nectar loaders
```

## Quickstart

The easiest way to run a cross-family panel is a single
[OpenRouter](https://openrouter.ai) key — one account, hundreds of models:

```python
from twelve_angry_llms import (
    Judge, Panel, PreferenceDatapoint, ScoringProtocol,
    OpenRouterClient, to_records, to_jsonl,
)

openrouter = OpenRouterClient()  # reads OPENROUTER_API_KEY
panel = Panel([
    Judge(model="openai/gpt-4o-2024-08-06", client=openrouter),
    Judge(model="anthropic/claude-sonnet-4.5", client=openrouter),
    Judge(model="qwen/qwen-2.5-72b-instruct", client=openrouter),
])

datapoints = [
    PreferenceDatapoint(
        prompt="What causes tides?",
        responses=(
            "The gravitational pull of the moon and sun.",
            "Mostly wind patterns over the ocean.",
            "The moon's gravity, with a smaller solar contribution.",
        ),
    ),
]

results = panel.annotate_sync(datapoints, ScoringProtocol())
print(results[0].ija)              # panel agreement on this datapoint, in [-1, 1]
print(panel.diagnostics(results))  # Krippendorff's alpha, judge-judge correlations

to_jsonl(to_records(results), "annotated.jsonl")  # TRL schema + IJA columns
```

Judges aren't tied to OpenRouter: `OpenAICompatibleClient` (which
`OpenRouterClient` is a preconfigured version of) speaks to any
OpenAI-compatible endpoint — OpenAI directly, Together, Groq, a local vLLM
or Ollama server, ... — via its `base_url`. Keys are read from environment
variables; pass `api_key_env` to point at a different variable per
provider.

```python
from twelve_angry_llms import OpenAICompatibleClient

openai = OpenAICompatibleClient()  # OPENAI_API_KEY
local = OpenAICompatibleClient(base_url="http://localhost:11434/v1", api_key="-")
```

Each exported record carries:

| column | meaning |
|---|---|
| `prompt`, `chosen`, `rejected` | the standard TRL/DPO interface |
| `prompt_ija` | panel agreement over all K candidates (Kendall's τ-b) |
| `pair_agreement` | fraction of judges preferring chosen over rejected — usable directly as a soft label for conservative DPO / soft-label methods |
| `judge_values` | raw per-judge scores or rankings |

## Two elicitation protocols

- **`ScoringProtocol`** — every judge scores each response 1–5 against a
  shared rubric (the UltraFeedback style). Also yields per-judge score
  margins for free.
- **`RankingProtocol`** — every judge orders all K responses best-to-worst
  in one pass (the Nectar style).

Both parse into per-response utilities, so IJA and all exports work
identically. Judges receiving one shared guideline per protocol is the
point: any disagreement then reflects the judges, not prompt wording.

## Command line

```bash
tal annotate --input data.jsonl --panel panel.yaml --output annotated.jsonl
tal export   --input annotated.jsonl --output pairs.jsonl
```

`data.jsonl` rows need `prompt` and `responses`; `panel.yaml` declares the
judges and (optionally) a response cache:

```yaml
metric: kendall
cache: .tal/cache.sqlite
judges:
  - model: openai/gpt-4o-2024-08-06
    provider: openrouter                 # the default
  - model: anthropic/claude-sonnet-4.5
    provider: openrouter
  - model: qwen/qwen-2.5-72b-instruct
    provider: openrouter
  - model: llama3.1:70b                 # any OpenAI-compatible endpoint also works
    provider: openai
    base_url: http://localhost:11434/v1
```

With the `data` extra you can annotate the raw (non-binarized) datasets
directly: `tal annotate --dataset ultrafeedback --limit 1000 ...`.

### Reproducible OpenRouter runs

One caveat with aggregators: they load-balance open-weight models across
hosts with different quantizations, which can silently change judge
behavior between runs. Pin the routing with `extra_body` (merged into
every request, and part of the response-cache key):

```yaml
  - model: qwen/qwen-2.5-72b-instruct
    provider: openrouter
    extra_body:
      provider: {order: [together], allow_fallbacks: false, quantizations: [fp16]}
```

Use exact-versioned model slugs (not `~latest` aliases) for the same
reason. See `research/configs/panel.example.yaml` for a fully pinned
five-judge panel.

## Reproducibility

- Temperature defaults to 0 everywhere.
- `CachedClient` / the `cache:` key stores every raw response in SQLite,
  keyed on (model, sampling settings, messages) — re-runs never re-bill.
- Clients track token usage (`client.usage`) so runs can report cost.

## The research

This library is the instrument for an ongoing study of IJA as a
label-reliability signal — hypotheses, judging protocol, and experimental
phases live in [research/RESEARCH_PLAN.md](research/RESEARCH_PLAN.md).
Working title: *"Twelve Angry LLMs: Judge Agreement as a Label-Reliability
Signal for Preference Data."*

## License

MIT
