Metadata-Version: 2.4
Name: spec-eval
Version: 0.2.1
Summary: A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code.
Author: Ben Jones
License: MIT
Project-URL: Homepage, https://github.com/benjaminjones/spec-eval
Project-URL: Repository, https://github.com/benjaminjones/spec-eval
Project-URL: Issues, https://github.com/benjaminjones/spec-eval/issues
Keywords: spec-driven-development,documentation-drift,docs-as-code,living-documentation,llm-as-judge,code-documentation,backfill-documentation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anthropic>=0.40
Requires-Dist: openai>=1.50
Requires-Dist: google-genai>=1.0
Requires-Dist: PyYAML>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

# spec-eval — Build fast with AI. Validate with specs.

**Your code changes fast. Your docs fall behind. spec-eval keeps them honest.**

*The trust layer for [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development) — iterate fast, keep the spec and code in agreement.*

*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*

[Get started](#try-it) · [Tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)

A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
trusting. How far you lean on it is up to you. **No specs yet? spec-eval writes them for you.**

## Table of contents

- [The problem](#the-problem)
- [What it checks](#what-it-checks)
- [What it creates](#what-it-creates)
- [Try it](#try-it)
  - [Run via prompt chat (no setup)](#run-via-prompt-chat-no-setup)
  - [Run in the terminal, no key (Claude subscription)](#run-in-the-terminal-no-key-claude-subscription)
  - [Run in the terminal with an API key (for CI)](#run-in-the-terminal-with-an-api-key-for-ci)
- [How the scores are made](#how-the-scores-are-made)
- [Reading the scores](#reading-the-scores)
- [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion)
- [Proof it works](#proof-it-works)
- [More](#more)
- [FAQ](#faq)
- [Feedback](#feedback)
- [Acknowledgements](#acknowledgements)

## The problem

Specs go stale the moment code moves on — and one stale spec is enough to stop a reader trusting all of them.
AI coding agents make code move faster than any hand-maintained doc can follow.

**spec-eval** treats specs like code: **run a check — via a chat prompt, an ordinary command, or a CI step — and
the spec is measured against the source.** A spec you can check is a spec you can trust.

## What it checks

Three simple questions:

| Check | The question it answers | How |
|---|---|---|
| **coverage** | Does this code have a spec at all? | Just counts files. Free — no AI, no key. |
| **drift** | Does the spec say something the code doesn't do? | An AI reads each spec and its code and lists the clashes. |
| **sufficiency** | How completely does the spec capture the code's behavior? | An AI lists what the spec misses and scores it 0–1 — an indicator, not a guarantee. |

Your **code is the source of truth** — the spec is what gets graded against it.

## What it creates

Everything it writes is plain markdown, right next to your code:

```text
 your code                 specs beside it              spec-reports/
┌────────────┐             ┌────────────┐             ┌────────────────┐
│ parser.py  │  generate   │ parser.md  │  coverage   │ coverage.md    │
│ api.py     │ ──────────► │ api.md     │  audit      │ report.md      │
│ …          │             │ …          │ ──────────► │ sufficiency.md │
└────────────┘             └────────────┘ sufficiency └───────┬────────┘
                                                              │
                                                              ▼
                                                       SPEC-HEALTH.md
                                                    (one-page scorecard)
```

## Try it

Prefer a guided walkthrough? **[Getting-started tutorial →](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md)**

**The commands** — in the order you'll typically use them:

- **coverage** — which files have no spec yet? *(free — no AI)*
- **generate** — writes a plain-English spec beside each file that has none *(uses AI)*
- **sufficiency** — how completely does each spec capture the code? Scored 0–1 *(uses AI)*
- **audit** — the drift check: does any spec clash with the code? Run it at every milestone *(uses AI)*

None of them change your files — reports go to `spec-reports/`. The exception is `generate`, which writes new
specs beside your code: review them and commit the keepers like code.

The `(uses AI)` commands (`generate`, `sufficiency`, `audit`) need an AI behind them — `coverage` never does.
Three ways to run them, least setup first:

### Run via prompt chat (no setup)

Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
land in the chat. Two skills, one per job — point your agent at the right one:

**Write specs** — the [authoring skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) carries the spec structure
(the `INV-*` invariant and `AC-*` acceptance-criteria tables come from its rubric and templates):
```text
Read path/to/spec-eval/skills/spec-authoring/SKILL.md — and the templates
in its templates/ folder — and follow it: write specs for my code.
```

**Check specs** — the [checking skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) grades what exists (drift + sufficiency):
```text
Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
first check coverage, then check my specs against my code.
```

> [!NOTE]
> `path/to/spec-eval` is wherever you cloned this repo. No clone? Give your agent the skill's link by name:
> [`spec-authoring/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) to write specs,
> [`spec-check/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) to check them.

> [!TIP]
> **Author and check in separate sessions.** A checker that still remembers its authoring decisions tends to
> echo them instead of re-reading the code — the same reason a PR isn't reviewed by its author. Write specs in
> one session; open a fresh one to check them.

Optionally save reports — add:
```text
and save the results to spec-reports/
```
The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).

**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](https://github.com/benjaminjones/spec-eval/tree/main/skills/) into your agent's skills
folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
```bash
mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
```
```text
Which files have no spec yet?        # coverage
Write a spec for src/parser.py       # generate
Check my specs against my code       # sufficiency + audit
```

### Run in the terminal, no key (Claude subscription)

Install once, then add `--model claude-code` to any `(uses AI)` command — **spec-eval** routes through the
`claude` CLI you're already logged into (no key; your `ANTHROPIC_API_KEY` is hidden from the call, so it can't
accidentally bill the paid API). The same four commands, in the same order as the list above:
```bash
pip install spec-eval                                     # or, from a clone:  pip install -e .
spec-eval coverage    ./your-project                      # free — no AI, no key
spec-eval generate    ./your-project --model claude-code
spec-eval sufficiency ./your-project --model claude-code
spec-eval audit       ./your-project --model claude-code
```
Every terminal run writes its reports to `spec-reports/` (plus a `runs.jsonl` history line).

### Run in the terminal with an API key (for CI)

Export a key and run any command above **without** `--model claude-code` — an API key is the default. This is
the option for CI, the checks that run automatically on every push (store the key as a repo secret):
```bash
export ANTHROPIC_API_KEY=sk-ant-...     # or OPENAI_API_KEY / GOOGLE_API_KEY
```

> [!NOTE]
> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
> `sufficiency` when you've added behavior. The [tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) has step-by-step guides for
> starting with no specs and for existing docs.

> [!TIP]
> **Version control is also your preview.** The three checks never modify your project; `generate` adds
> ordinary new files — review them with `git diff`, edit what's off, and drop any you don't want with
> `git checkout -- <file>`. (A full `generate --overwrite` rewrites everything, so save it for first setup.)

A few things worth knowing:

- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
  Prefer one spec per folder? `--layout per-dir` ([FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
  (Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
  READMEs are left alone. (Folder specs and overview indexes only appear if you opt in with `--layout` / `--overview`.)
- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)

**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
code always needs a spec; copy-paste setup in [the tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine):
```bash
spec-eval coverage . --min 90    # fails the commit if coverage drops below 90%  (free — no AI)
```

## How the scores are made

Three scores, three simple ideas:

- **coverage** = spec files you *have* ÷ spec files you *need* → a percent. Just counting, **no AI**.
  *(9 of 10 code files have a spec → 90%.)*
- **drift** = the number of spots where the spec and the code flat-out disagree. **`0` = they match.**
- **sufficiency** = how much of what the code does is actually written in the spec, from **`0` to `1`**.
  *(`1.0` = it's all there; `0.1` = almost none of it is.)*

Those three are **per spec**. The repo's one **sufficiency** score is just their **average** — each spec's
`0`–`1`, added up and divided by how many specs were scored. *(coverage is already one repo-wide `%`; drift is
the total count across specs.)*

`coverage` is pure counting. The other two need an **AI reader**: it reads each spec next to its code and grades
the pair against a fixed rubric — one model call per pair. Two things follow from that:

> [!NOTE]
> AI scores **wobble**: the same spec might get 0.78 one run and 0.72 the next, like two teachers grading the same
> essay. Watch the **trend**, not the decimal — a jump like 0.72 → 0.85 is a real improvement.

> [!IMPORTANT]
> AI calls **spend tokens** — through your API key, your Claude subscription with `--model claude-code`, or
> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.

**Want the exact rules behind each score?** spec-eval specs its own scoring — read
[`spec_eval/coverage.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/coverage.md) (how coverage counts),
[`spec_eval/rubric.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/rubric.md) (the drift rules), and
[`spec_eval/sufficiency.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/sufficiency.md) (how the 0–1 score is decided).

## Reading the scores

Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):

- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/coverage.md))*
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/report.md))*
- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
  (`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/sufficiency.md))*

The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md).

Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
time.

## Higher stakes? Get a second opinion

`drift` and `sufficiency` are **AI judgments** — one model can miss something or be too generous. For a spec that needs extra attention — a critical path, complex design, or compliance code — run the
**same check with two different AI vendors** and compare. Give each run its own
`--out` folder (otherwise the second run overwrites the first report):

```bash
spec-eval audit ./your-project --model anthropic:claude-opus-4-8 --out spec-reports/claude
spec-eval audit ./your-project --model google:gemini-3.5-flash   --out spec-reports/gemini

diff spec-reports/claude/report.md spec-reports/gemini/report.md   # or read them side by side
```

How to read two reports:

- **Both flag it** → almost certainly real. Fix these first.
- **Only one flags it** → a judgment call. Read the quoted evidence and decide.
- **Sufficiency scores differ a little** → normal wobble. Compare the **gaps lists**, not the decimals — the same
  missing behavior named by both graders is the signal.

Each vendor reads its own key from the environment — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY` —
so put both keys in one `.env` and each run picks up the one it needs. (Want one fewer key? Swap either side for
`--model claude-code`, which uses your Claude Code login instead.)

## Proof it works

- **spec-eval specs itself** — every module in [`spec_eval/`](https://github.com/benjaminjones/spec-eval/tree/main/spec_eval/) has its own spec:
  **coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md) — the exact, dated score).

## More

- **Works with [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development).** SDD's goal is a spec you can *trust*, and **spec-eval** is that trust layer.
  Use it three ways: **grade the specs you already have**, **generate specs** for code that has none, or **check as
  you write** (with the skills). Spec-first, code-first, or in between — it meets you where you are.
- **Grow the spec and the code together.** The best version of "in between": draft a little spec, write a little
  code, check they still agree — repeat. The two can even move in parallel (you shape the spec while your coding
  agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
  afterward.
- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
  Action can run the checks for you — see [Make it routine](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine).
- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.

## FAQ

Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
**[FAQ.md](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)**.

## Feedback

Found a bug, a bad score, or a spec that misses the point? [Open an issue](https://github.com/benjaminjones/spec-eval/issues) — a report snippet and the command you ran is plenty.

## Acknowledgements

Built on the Anthropic, OpenAI, and Google SDKs (MIT / Apache-2.0) + PyYAML.

---
*Terms you may be searching for: software specification · documentation drift · living documentation · docs as code · backfill
documentation · spec-driven development (SDD) · agentic development · AI-assisted coding · vibe coding · vibe engineering ·
AI code documentation · LLM-as-judge / trust eval. MIT licensed.*
