Metadata-Version: 2.4
Name: pluto-llm-diff
Version: 0.1.0
Summary: git diff, but for LLM behavior — behavioral regression testing for language models.
Project-URL: Homepage, https://github.com/SiddharthFX/llm-diff
Project-URL: Repository, https://github.com/SiddharthFX/llm-diff
Project-URL: Issues, https://github.com/SiddharthFX/llm-diff/issues
Project-URL: Whitepaper, https://github.com/SiddharthFX/llm-diff/blob/main/WHITEPAPER.md
Author: Pluto AI Research Lab
License-Expression: MIT
License-File: LICENSE
Keywords: benchmarking,cli,diff,evaluation,llm,ollama,openai,regression-testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: requests>=2.28
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == 'dev'
Description-Content-Type: text/markdown

# llm-diff ⚡

> **git diff, but for LLM behavior.**

[![CI](https://github.com/SiddharthFX/llm-diff/actions/workflows/ci.yml/badge.svg)](https://github.com/SiddharthFX/llm-diff/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-diff)](https://pypi.org/project/llm-diff/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

![llm-diff demo](docs/demo.gif)

You swapped `gpt-4o` for `gpt-4o-mini` to cut costs. You upgraded Llama 3.1 to 3.2 because the benchmarks went up. **Did your model's *behavior* actually get better — or did it just get chattier, start ignoring your formatting rules, and flip its answers when you rephrase a question?**

Benchmarks measure capability. `llm-diff` measures **behavior** — and shows you exactly what changed, in the format every engineer already speaks: a diff.

```
- baseline   ollama/qwen2.5:0.5b
+ candidate  openai/gpt-4o-mini
```

One command. Three behavioral dimensions. Colored deltas. CI-friendly exit codes.

## Install

```bash
pip install llm-diff
```

## Quickstart

Compare a local Ollama model against an OpenAI model:

```bash
export OPENAI_API_KEY=sk-...
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-mini
```

Illustrative output (`baseline` vs `candidate` shown for clarity):

```
- baseline   ollama/baseline-model
+ candidate  openai/candidate-model

╭──────────────────────┬──────────────────┬──────────────┬──────────────┬────────╮
│ Dimension            │ Metric           │ - baseline   │ + candidate  │      Δ │
├──────────────────────┼──────────────────┼──────────────┼──────────────┼────────┤
│ Instruction Fidelity │ fidelity score   │ 1.00         │ 0.50         │  -0.50 │
│ Verbosity Profile    │ total words      │ 41           │ 118          │    +77 │
│                      │ preamble words   │ 0            │ 9            │     +9 │
│                      │ markdown headers │ 0            │ 3            │     +3 │
│ Reasoning Consistency│ consistency      │ 1.00         │ 1.00         │   0.00 │
│ Latency              │ avg latency (s)  │ 0.84         │ 1.92         │  +1.08 │
╰──────────────────────┴──────────────────┴──────────────┴──────────────┴────────╯
        Δ = candidate − baseline · green improvement · red regression
```

More recipes:

```bash
# Two local models, fully offline
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b

# Any OpenAI-compatible server: vLLM, LM Studio, llama.cpp, TGI, ...
llm-diff openai/my-finetune openai/my-base \
  --openai-base-url http://localhost:8000/v1

# Open-weight models via Hugging Face Inference Providers (OpenAI-compatible)
llm-diff "openai/meta-llama/Llama-3.3-70B-Instruct" "openai/Qwen/Qwen3-8B" \
  --openai-base-url https://router.huggingface.co/v1 --api-key $HF_TOKEN

# Machine-readable report for dashboards and CI artifacts
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-mini --json > diff.json

# Audit the raw responses behind every score
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b --show-responses
```

Model specs are `backend/model`. Unprefixed specs fall back to `--backend` (default: `ollama`), and only the *first* slash is parsed — so HuggingFace-style ids like `meta-llama/Llama-3.3-70B-Instruct` pass through intact.

## The three dimensions (MVP)

| Dimension | The question it answers | How it's scored |
|---|---|---|
| **Instruction Fidelity** | Does the model obey strict formatting constraints? | Probe demands *exactly 3 bullets, no introduction*. +0.5 for exactly 3 bullet lines, +0.5 for a clean start with zero preamble. |
| **Verbosity Profile** | How much does the model say, and how much of it is filler? | One open-ended explanation, measured three ways: total words, filler-preamble words ("Certainly! Here is…"), and markdown header count. |
| **Reasoning Consistency** | Does the same logic get the same answer when reworded? | Two isomorphic syllogisms with nonsense words (memorization-resistant). Score 1.0 if the Yes/No verdict survives the reframing, 0.0 if it flips. |

Every diff runs at `temperature=0` by default, so results are as reproducible as the backends allow.

## Built for CI

`llm-diff` speaks exit codes: `0` all probes succeeded, `1` fatal config error, `2` usage error, `3` partial diff (some probes failed — shown as `ERR` rows, never a crash). Transient failures (timeouts, 429s, 5xx) are retried with exponential backoff; hard failures surface immediately with actionable messages.

```yaml
# .github/workflows/model-swap.yml (sketch)
- run: llm-diff ollama/current-prod ollama/candidate --json > diff.json
- run: python scripts/check_thresholds.py diff.json   # gate on deltas
```

Native `--fail-if "fidelity<0.8"` gating is on the roadmap below.

## Roadmap: from 3 dimensions to 7

The MVP battery is deliberately small — a fast smoke test, not a benchmark. The v1.0 goal is **seven dimensions** with a much larger, community-built probe set:

| # | Dimension | Status |
|---|---|---|
| 1 | Instruction Fidelity | ✅ shipped |
| 2 | Verbosity Profile | ✅ shipped |
| 3 | Reasoning Consistency | ✅ shipped |
| 4 | **Hallucination Propensity** — does the model invent facts, citations, and APIs under pressure? | 🔜 planned |
| 5 | **Refusal Sensitivity** — does the model over-refuse benign requests, and did that change between versions? | 🔜 planned |
| 6 | **Domain Retention** — did niche knowledge (SQL dialects, legal terms, medical vocab) survive the new release? | 🔜 planned |
| 7 | **Sycophancy Drift** — does the model cave and change correct answers when the user pushes back? | 🔜 planned |

Also planned for v0.2: custom probe packs via YAML (`llm-diff a b --probes my_domain.yaml`), N-way comparison matrices, a native `hf/` backend prefix for Hugging Face Inference Providers, and `--fail-if` threshold gating for CI.

## Contributing

**The most valuable thing you can contribute is a probe set.** The three built-in probes prove the mechanism; the community can make it comprehensive. If you know how models *should* behave in your domain — medicine, law, code review, customer support, your native language — you can encode that knowledge as probes + scorers, no ML expertise required.

Great first contributions:

1. **Probe packs** — propose a set of prompts + expected behaviors for one dimension (open an issue with the `probe-pack` label to discuss the format).
2. **Scorers** — every scorer is a pure function of response text in [`llm_diff/scorers.py`](llm_diff/scorers.py), trivially unit-testable. Sharpen the heuristics or add new ones.
3. **Backends** — the transport layer in [`llm_diff/core.py`](llm_diff/core.py) is ~100 lines per backend.

Dev setup:

```bash
git clone https://github.com/SiddharthFX/llm-diff
cd llm-diff
pip install -e ".[dev]"
pytest   # fully offline — tests run against a bundled mock backend
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. Adding a dimension end-to-end touches exactly three places: a prompt in `probes.py`, a `score_*` function in `scorers.py`, and one hook in `evaluate()` — the table, summary, JSON report, and exit codes pick it up automatically.

## Research

This tool exists to answer a bigger question: **do successive LLM releases trade behavioral fidelity for benchmark scores?** Read the study design in [WHITEPAPER.md](WHITEPAPER.md) — and help us run it.

## License

[MIT](LICENSE) © 2026 Pluto AI Research Lab
