Metadata-Version: 2.5
Name: skill-harness
Version: 0.3.0
Summary: I wanted to know if you could tell if a skill was any good. This runs the same task with the skill and without it, and reports what can honestly be said about the difference - often "not enough to call it". A missing figure is a typed refusal, never an invented score. First-class Claude Code support.
Project-URL: Homepage, https://github.com/MrBinnacle/skill-harness
Project-URL: Repository, https://github.com/MrBinnacle/skill-harness
Project-URL: Issues, https://github.com/MrBinnacle/skill-harness/issues
Project-URL: Changelog, https://github.com/MrBinnacle/skill-harness/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/MrBinnacle/skill-harness/tree/main/docs
Author: MrBinnacle
License: MIT
License-File: LICENSE
Keywords: ab-testing,agent-skills,ai-agents,claude-code,eval-harness,evals,llm-evaluation,prompt-evaluation,skill-evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: anthropic>=1.2.0
Requires-Dist: click>=8.1
Requires-Dist: openai<4,>=2.41
Requires-Dist: pydantic>=2.6
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.11
Requires-Dist: statsmodels>=0.14
Requires-Dist: tiktoken<1.0,>=0.7
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: jsonschema>=4.26.0; extra == 'dev'
Requires-Dist: mypy>=2.3.1; extra == 'dev'
Requires-Dist: numpy>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=4; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest-randomly>=4.1; extra == 'dev'
Requires-Dist: pytest-socket>=0.8.1; extra == 'dev'
Requires-Dist: pytest-xdist>=3.5; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: scipy>=1.11; extra == 'dev'
Requires-Dist: statsmodels>=0.14; extra == 'dev'
Provides-Extra: inspect
Requires-Dist: inspect-ai>=0.3.245; extra == 'inspect'
Requires-Dist: inspect-swe>=0.2.65; extra == 'inspect'
Provides-Extra: sitegen
Requires-Dist: jsonschema>=4.26.0; extra == 'sitegen'
Description-Content-Type: text/markdown

<p>
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/MrBinnacle/skill-harness/main/assets/banner-dark.svg">
    <img alt="skill-harness — the skill eval that refuses to invent a score" src="https://raw.githubusercontent.com/MrBinnacle/skill-harness/main/assets/banner-light.svg" width="680">
  </picture>
</p>

# skill-harness

[![CI](https://github.com/MrBinnacle/skill-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/MrBinnacle/skill-harness/actions/workflows/ci.yml)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/MrBinnacle/skill-harness/blob/main/LICENSE)
[![PyPI](https://img.shields.io/pypi/v/skill-harness.svg)](https://pypi.org/project/skill-harness/)

## Why this exists

I wanted to know if you could tell if a skill was any good.

That sentence is the owner's, and it is the question this repository was built to answer. The
longer account, and the two commands that re-derive how much machinery the question cost, are
in [docs/why-this-exists.md](https://github.com/MrBinnacle/skill-harness/blob/main/docs/why-this-exists.md).

A skill is a file an agent loads. Search for one that makes AI writing read less like AI
writing and a dozen come back. Each costs context in every conversation, whether or not it
fires. Reading the file tells you how it reads. It does not tell you whether it changes an
outcome.

skill-harness runs the same task with the skill and without it, and reports what the evidence
supports about the difference. The most common report is "not enough to call it."

The skills it screens live in [MrBinnacle/skills](https://github.com/MrBinnacle/skills). Claude
Code skills are the first-class subject; the subject layer is built to take other agent
ecosystems.

## What does this skill cost you, and which parts of it are worth that cost?

That is the ratified wording. "Is this skill good" hides two questions. A skill has a
**price**, paid in every conversation whether or not it fires. It has a **benefit** that may
or may not appear when it does. The price is arithmetic on text and costs nothing to report.
The benefit needs a paid comparison, and the evidence for it usually stops short of a call.

The two are measured differently, refused differently, and reported in separate fields. The
instrument does not collapse them into one score.

## Try the free offline skill audit

`skill audit` runs offline. No API key, no database, no network.

```bash
pip install skill-harness
skill-harness skill audit path/to/your/SKILL.md
```

It reports three properties. The **cost triple**: what the skill costs standing, fired, and in its
side docs (aux), as arithmetic on text. A set of **structural checks** against
[Anthropic's authoring spec](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices).
An **evaluability preflight**: what a paid run could and could not measure about this skill
today.

The output below is from that command on the committed fixture
`tests/fixtures/sers/declared-synthetic-positive-control/SKILL.md`, run on 2026-09-06 at
v0.3.0 in a 100-column terminal. Two lines are removed: `source:` (a local absolute path) and
`sha256:`, and the Summary line's trailing pointer to `docs/concepts/why-unmeasured.md` is
dropped so the line renders whole. Everything else is verbatim.

```text
OFFLINE AUDIT — no API calls, no cost
  skill:  declared-synthetic-positive-control
  body:   9 lines / 41 words · frontmatter keys: description, name
                               Structure (Anthropic authoring spec)
┌────────┬───────────────────────────────────┬────────────────────────────────────────────────────┐
│ Level  │ Check                             │ Finding                                            │
├────────┼───────────────────────────────────┼────────────────────────────────────────────────────┤
│ PASS   │ name                              │ name 'declared-synthetic-positive-control' meets   │
│        │                                   │ spec                                               │
│ INFO   │ description-unparsed-block-scalar │ description uses a multi-line YAML block scalar,   │
│        │                                   │ which this audit's minimal frontmatter parser      │
│        │                                   │ cannot read — description content checks skipped   │
│        │                                   │ (UNMEASURED, not passed)                           │
│ PASS   │ body-length                       │ body 9 lines (budget 500)                          │
│ WARN   │ standing-cost-unparseable         │ router listing line (frontmatter name +            │
│        │                                   │ description) is not readable as a single-line pair │
│        │                                   │ — standing cost UNMEASURED (no number; a silent    │
│        │                                   │ default would understate the per-turn tax)         │
└────────┴───────────────────────────────────┴────────────────────────────────────────────────────┘
Evaluability preflight — what a paid run could measure today:
  Tier-1 mechanical axes: citation_presence_per_flag, compliance_proxy, hedge_index,
structure_score, verbosity (style-shaped only).
  Behavior-shaped claims (correctness, tool use, outcomes): no mechanical instrument in v0.1 → the
recorded state would be UNMEASURED, not an estimate.
  Judge-graded axes: require a calibrated (judge, axis) pair — none exists in a fresh install →
UNMEASURED until you run `calibrate`.
  Standing cost (mechanical): UNMEASURED — frontmatter could not be parsed well enough to count the
router listing line (see standing-cost-unparseable).
  Fired cost (mechanical): raw 68 tokens · calibrated 77 tokens (x1.128, measured range
1.084-1.179) -- skill body charged when the skill runs and its body is read.
  Aux cost (mechanical): raw 0 tokens · calibrated 0 tokens (x1.128, measured range 1.084-1.179) --
other documentation files beside the skill (progressive disclosure); 0 when the skill directory has
none.

Summary: 2 pass · 1 warn — UNMEASURED is a recorded state, not a failure.
Clause evidence: UNMEASURED (no_extraction: clause evidence requires an extraction output; produce
one with skill init --out <file>)
```

The parser could not read the description, so the audit says so and skips the check. It does
not pass a field it never read. The standing cost has no number for the same reason, and the
audit prints `UNMEASURED` rather than a default.

`--strict` exits 1 on warnings, for CI. From v0.3.0 the CLI sets UTF-8 on its own output
streams; on v0.2.3 a Windows console needs `PYTHONUTF8=1` set first or the command exits 1 with
`UnicodeEncodeError`.

## What it has found so far

**Zero production-skill KEEPs.** The full keep lane has fired end to end once, on 2026-07-27.
The subject was a *declared synthetic positive control*: a skill written to carry an invented
fact, so the effect exists by construction. It returned KEEP at 8/8 with the skill against 0/8 without,
posterior probability of a win 0.99
([SERS receipt, 2026-07-27](https://github.com/MrBinnacle/skill-harness/blob/main/docs/sers/receipts/synthetic-control-keep-2026-07-27.json)).
That run shows the instrument fires when an effect is present. It says nothing about whether
any real skill is worth its slot.

The most common result is that the model already does the task without a skill. On two
deliberately hardened tasks a frontier agent passed 14 of 14 no-skill runs. Nothing was left
for a skill to improve, so nothing could be measured. That is a finding about the task, and it
is written up in
[the double-ceiling case study](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/double-ceiling-structurally-unmeasured.md).

The paired run in that study, July 2026, cost about $6.17 and returned the pre-registered
NO-GO: an apparatus check, not a measurement of benefit. The receipt records it as one
([`double-ceiling-nogo-2026-07-09.json`](https://github.com/MrBinnacle/skill-harness/blob/main/docs/sers/receipts/double-ceiling-nogo-2026-07-09.json)).

None of that is a scheduling accident. A sized benefit run launches only on the first task
whose no-skill screen returns a pass rate below 1. Every production skill screened so far
ceilings at 1: the model passes every attempt without it
([observation ledger](https://github.com/MrBinnacle/skill-harness/blob/main/docs/observations/README.md)).

## Why it refuses

Comparing a skill against nothing is noisier than it looks. On a 60-trial arc of identical
agentic coding tasks, run-to-run output-token variation measured CV ≈ 17.6% (RMS across
cells; mean-of-cells 14.6%, median 10.4%) on an Opus-class model
([findings record](https://github.com/MrBinnacle/skill-harness/blob/main/docs/findings/why-naive-skill-benchmarks-mislead.md)).
At that coefficient of variation, a three-runs-a-side comparison cannot separate differences
under roughly 30–40% from noise, and three runs a side is what most published skill
comparisons use. Hand-picked tasks tilt the result before anything runs. Pass/fail test banks
price what a skill *costs* and skip what it *does*. The findings record carries an evidence
grade on each claim.

The design rule follows from that. **A figure that is not there is stated as a typed refusal,
never filled in.** No placeholder zero, no free-typed excuse, no estimate standing in for a
measurement.

Three consequences follow from the rule.

**One.** Every paid comparison has a control arm. With and without, never a score in a vacuum,
because a score in a vacuum cannot show that the model did not need the skill.

**Two.** When the evidence cannot carry a call, the answer is `UNMEASURED` with a reason from a
fixed list of eight: `no_data`, `inadmissible`, `underpowered`, `falsifying_case_missing`,
`budget_exhausted`, `falsifying_case_stale`, `fdr_correction_failed`, `mechanical_vacuous`
(`src/skill_harness/aggregation/status.py`). Which kind of not-knowing is more information
than not-knowing alone. Definitions:
[`docs/concepts/why-unmeasured.md`](https://github.com/MrBinnacle/skill-harness/blob/main/docs/concepts/why-unmeasured.md).

**Three.** Evidence passes a gate before it enters an aggregate, and the gate result is
snapshotted at write time in an append-only store. Data that fails the evidence-admissibility
gate is kept and never counted. A judge-graded result counts only where that judge has been
calibrated on that axis first. Calibration swaps answer order to cancel position bias,
controls for length, defends against injection, and measures agreement with a human.

The same rule points inward. Two of the instrument's own weak points are measured, and both
numbers stay on the front page:

**Extraction repeat-variance:** MEASURED for one skill — three repeat extractions of the same
`SKILL.md` returned 29/33/34 clauses, so clause counts are **not stable** run to run and
nothing downstream is allowed to key on clause position
([#152](https://github.com/MrBinnacle/skill-harness/issues/152)).

**Vacuity-flag precision:** MEASURED at 0.972 by blind cross-family adjudication over 106
adjudicated rows, and that figure is **flag-level only**
([#153](https://github.com/MrBinnacle/skill-harness/issues/153)). When the adjudicators also
had to agree on *which kind* of vacuity, kind-precision 0.835: `not_a_directive` matched 77/77,
while `weak_directive` matched 4/20. The vacuity-flag detector's recall is UNMEASURED: the
unflagged clauses were never adjudicated.

## What it measures, and what it refuses to

The answer comes back as one of three verdicts: **KEEP**, **CUT**, or **CAN'T-TELL-YET**.
Which of those a skill is eligible for depends on its registered value class, not on the
numbers alone. A CUT says why: `subsumed` (the model was already doing it), `no_lift` (the
model needed help and the skill did not deliver it), or `harmful`.

The **value-class guard** sits on that. A skill can exist to stop one specific wrong move. A
model that passes without such a skill has not shown the skill is useless; it has shown the
trap did not come up. So `subsumed` is a CUT only for skills registered as
`TRANSFORMATIVE_LIFT`, the class whose whole claim is lift above the bar. Every other class
reclassifies to CAN'T-TELL-YET, because this is the wrong instrument for that kind of skill,
not a verdict on it.

Two skills from the collection moved that way when the guard landed: `append-only-evidence-design`
(calibration) and a hardened `git-pull-rebase-trap` (trap-discipline). Under the pre-guard rule
both returned **CUT (subsumed)**, each at a no-skill pass rate of 1.00; the value-class guard
reclassified both to CAN'T-TELL-YET
([receipts](https://github.com/MrBinnacle/skill-harness/tree/main/docs/sers/receipts/)). The
pre-guard CUTs stay in the record as dated output, not edited into agreement.

The other half has never fired: a paired run sizing how much a skill helps once the model is
known to need help. By design, a sized benefit run launches only when a screen returns a
sub-1 pass rate, and none has.

## Measuring for real

```bash
skill-harness skill init path/to/SKILL.md --execute   # extract testable claims
skill-harness run ablation <skill_id> --execute       # the with/without comparison
skill-harness run evaluate-skill <skill_id>           # aggregate to a verdict
```

`ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY`. Every `run` subcommand is dry-run by default;
`--execute` is required to spend, and a per-run cap and a daily cap sit on top.
`skill init` is the exception: clause extraction is a model call in both modes, and
`--execute` decides only whether the result is persisted to the evidence DB. Without a key
it exits 1 before any call.
Reproduction scripts:
[`examples/`](https://github.com/MrBinnacle/skill-harness/tree/main/examples/).

## The reporting vocabulary is a published standard

The **Skill Efficacy Reporting Standard (SERS)** fixes the vocabulary: verdicts, refusal
reasons, the cost triple, the evidence-admissibility statuses, and the instrument identity.
Instrument identity is the model pin and prompt fingerprint that stamp *which generation*
produced a figure. SERS is a JSON Schema plus a prose companion, in
[`docs/sers/`](https://github.com/MrBinnacle/skill-harness/tree/main/docs/sers/).

SERS is separate from this tool's internals on purpose. Another harness can emit conforming
reports without adopting anything here. CI checks that this repository's own receipts validate
against the schema, that the schema's enums match the code's, and that deliberately poisoned
receipts are rejected.

Models change underneath every figure, so every figure has a shelf life. Instrument identity is
a required field: two numbers from two generations are visibly non-comparable rather than
averaged.

## What this isn't

It is not the most featureful skill benchmarker available.
If you want the most *featureful* skill benchmarking today,
[adewale's skill-eval-harness](https://github.com/adewale/skill-eval-harness) is the closest
neighbour and is further along on more axes. A few of its disciplines are on the adoption
list, with attribution. For comparing prompts and configurations rather than skills,
[promptfoo](https://github.com/promptfoo/promptfoo) is the mature choice. For evaluating models
and agents, [Inspect](https://github.com/UKGovernmentBEIS/inspect_ai) is the institutional one.

Reach for this one when the question is whether the number deserves to exist at all.

This repository makes no first-mover claim. The positioning was checked against primary sources
before it was written, and the first-or-only claims failed that check
([#39](https://github.com/MrBinnacle/skill-harness/issues/39)). Two claims carry the most
weight: the pre-spend eligibility gate, and the rule that thresholds are ratified from
enumerated tables rather than authored by hand. Both carry claim-status labels tied to an
external review plan ([#45](https://github.com/MrBinnacle/skill-harness/issues/45)). They
change by dated amendment, never silently.

## The other half

The verdicts land in a second repository:
[MrBinnacle/skills](https://github.com/MrBinnacle/skills), a small collection. There, each skill
carries its own dated evidence record and controlled results are read from that skill's record,
not a front-page roll-up. Skills are re-screened when a major model ships and publicly retired,
with the record intact, once the model no longer needs them or a platform change meets a
pre-registered trigger. Each retirement is made against its stated criterion.

The two repositories run on one rule, pointed at two different targets. This one does not state
a number the evidence does not support. That one does not keep a skill the evidence no longer
supports.

## Dig deeper

- [The receipts, rendered](https://mrbinnacle.github.io/skill-harness/) — the SERS receipts as a
  browsable site, one page per screened skill, cost triple beside the evidence grade. It renders
  the SERS instances only; the Markdown index below is the citable surface for every kind.
- [Measurement receipts index](https://github.com/MrBinnacle/skill-harness/blob/main/docs/receipts-index.md)
  — every case study, finding, observation, assurance report, ratification, SERS instance, and
  the `skill audit --extraction` join surface: what each claims and what each refuses to claim.
- [Why this exists](https://github.com/MrBinnacle/skill-harness/blob/main/docs/why-this-exists.md)
  — how a non-specialist ends up building a measurement instrument, and the loop that made it
  possible.
- [The double-ceiling case study](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/double-ceiling-structurally-unmeasured.md)
  — the run where there was nothing left to measure.
- [The ablation that caught its own author](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/ai-slop-sentinel-under-ablation.md)
  — three pre-spend catches before a contaminated result could ship.
- [When ablation measures the wrong layer](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/displaced-enforcement-skill-ablation-blind-spot.md)
  — if a discipline fires in a hook, ablating the skill text says nothing about the discipline.
- [`docs/PRD.md`](https://github.com/MrBinnacle/skill-harness/blob/main/docs/PRD.md) — the full
  specification: evidence model, oracle tiers, gate rules, CLI surface.
- [The observation ledger](https://github.com/MrBinnacle/skill-harness/blob/main/docs/observations/README.md)
  — per-record screen history, annotated rather than rewritten.

Status: v0.3.0 on PyPI. Not every older screen record is yet in the evidence store; the
observation ledger shows the evidence behind each record.

MIT licensed. Issues and PRs welcome:
[`CONTRIBUTING.md`](https://github.com/MrBinnacle/skill-harness/blob/main/CONTRIBUTING.md).
