Metadata-Version: 2.5
Name: pytest-skillcheck
Version: 0.2.0
Summary: Test agent skills against real coding agents: run a prompt, assert on what happened, grade the rest with an LLM.
Project-URL: Homepage, https://github.com/wiltaylor/pytest-skillcheck
Project-URL: Issues, https://github.com/wiltaylor/pytest-skillcheck/issues
Author-email: Wil Taylor <ai@wiltaylor.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: agent,claude,codex,eval,opencode,pytest,skills
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: pytest>=8
Description-Content-Type: text/markdown

# pytest-skillcheck

[![CI](https://github.com/wiltaylor/pytest-skillcheck/actions/workflows/ci.yml/badge.svg)](https://github.com/wiltaylor/pytest-skillcheck/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/pytest-skillcheck)](https://pypi.org/project/pytest-skillcheck/)

Test agent skills against real coding agents. Run a prompt, assert on what
actually happened, and grade the rest with an LLM.

A skill is a folder of instructions you hand to a coding agent. Nothing checks
that the agent still follows them after you edit the wording. skillcheck runs the
real CLI against a throwaway workspace and tells you.

```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["pytest-skillcheck"]
# ///
"""Tests for the git-graveyard skill."""

from skillcheck import main


def test_buries_a_public_repo_in_the_public_graveyard(run_skill):
    result = run_skill(
        "Bury ./deadproj",
        fake={"gh": {"wiltaylor/deadproj": {"visibility": "PUBLIC"}}},
        answers=["Yes, that graveyard is right. Go ahead."],
    )

    assert result.called("gh", "repo", "view", "wiltaylor/deadproj")
    assert not result.called("gh", "repo", "delete", "wiltaylor/deadproj")


if __name__ == "__main__":
    raise SystemExit(main(__file__))
```

Save that as `skills/git-graveyard/test.py`, `chmod +x`, and run it:

```sh
./skills/git-graveyard/test.py
```

uv fetches pytest-skillcheck, pytest runs, and the skill under test is the one the file
lives in — no `skill=` argument, no `conftest.py`.

## Contents

- [Install](#install)
- [Usage](#usage)
- [Contributing](#contributing)
- [Harnesses](#harnesses)
- [Settings](#settings)
- [Writing tests](#writing-tests)
- [Results](#results)
- [Commands](#commands)
- [Cost, and what CI can cover](#cost-and-what-ci-can-cover)
- [Adding a harness](#adding-a-harness)
- [Licence](#licence)

## Install

```sh
uv add pytest-skillcheck          # or: pip install pytest-skillcheck
skillcheck doctor          # which agent CLIs are installed, and what they would run
```

skillcheck is a pytest plugin, so its fixtures are available anywhere pytest runs
once it is installed.

## Usage

Scaffold a test beside a skill, then run it:

```sh
skillcheck new-test skills/my-skill    # writes skills/my-skill/test.py
chmod +x skills/my-skill/test.py
./skills/my-skill/test.py
```

The file knows which skill it covers from where it sits, so nothing tells it
twice. Each run appends to `results.json` beside it, and `skillcheck status
skills/` reads those back.

## Contributing

Pull requests are welcome, new harnesses most of all. Run what CI runs before you
open one:

```sh
uv run ruff check src tests
uv run ruff format --check src tests
uv run pytest tests -q
```

That suite covers parsing and the fake-binary machinery against recorded CLI
output. It cannot run a real agent, so anything you change in a harness needs a
skill test run by hand as well.

## Harnesses

| Harness | CLI | Skills installed into | Subagents | Isolated |
| --- | --- | --- | --- | --- |
| `claude` | `claude` | `.claude/skills/` | `.claude/agents/` | yes |
| `codex` | `codex` | `.agents/skills/` | `.agents/agents/` | yes |
| `opencode` | `opencode` | `.agents/skills/`, `.opencode/skills/` | `.opencode/agent/` | no |

By default skillcheck uses the first of `claude`, `codex`, `opencode` that is
installed. Override with `SKILLCHECK_HARNESS`:

```sh
SKILLCHECK_HARNESS=codex ./skills/my-skill/test.py
SKILLCHECK_HARNESS=claude,opencode ./skills/my-skill/test.py
SKILLCHECK_HARNESS=all ./skills/my-skill/test.py
```

A test can restrict itself with `@pytest.mark.harness("claude", "opencode")`, and
a harness whose CLI is missing skips rather than fails.

**Isolation.** An agent reads globally installed skills from your home directory,
and those *shadow* the copy under test — so without isolation you would be
grading whatever you last deployed. claude and codex are pointed at a throwaway
config directory with credentials symlinked in. opencode is not: its auth spans
several directories with no single file to link. Assert `not
result.reached_home()` to catch this.

## Settings

Environment first, `[tool.skillcheck]` in `pyproject.toml` second, defaults last.

| Variable | Default | Purpose |
| --- | --- | --- |
| `SKILLCHECK_HARNESS` | first installed | `claude`, `claude,codex`, or `all` |
| `SKILLCHECK_MODEL` | the CLI's own | model for every harness |
| `SKILLCHECK_MODEL_<HARNESS>` | — | per-harness model, e.g. `SKILLCHECK_MODEL_OPENCODE` |
| `SKILLCHECK_JUDGE` | `claude` | `claude`, `anthropic`, or your own callable |
| `SKILLCHECK_JUDGE_MODEL` | `sonnet` | model the judge grades with |
| `SKILLCHECK_TIMEOUT` | `900` | per-turn seconds |
| `SKILLCHECK_RESULTS` | next to the test | where results are written |
| `SKILLCHECK_RECORD` | `1` | set `0` to run without recording |

## Writing tests

### Assertions

`run_skill` returns a `RunResult`:

```python
result.exists("out.txt")          result.read("out.txt")     result.files()
result.ran(r"cargo build")        result.used_skill("name")  result.output
result.used_agent("reviewer")     result.delegated()         result.turns
result.called("gh", "repo", "view")   result.refusals("gh")  result.calls("gh")
result.acted_before_asking(r"--force")   result.reached_home()
```

### Skills that ask questions

`answers` replies as the agent hands control back. Each answer is the next user
message in the same session, and what the agent said at each point is kept in
`result.handbacks`.

```python
result = run_skill("Merge these repos", answers=["Yes, go ahead.", "No, keep the originals."])
```

Answers are sent whenever a turn ends, not only when the message looks like a
question: agents ask without question marks ("Confirm that I should proceed"),
and gating on punctuation strands the test. To check a skill stopped *before* it
acted, assert against the first turn's tool calls, which is exact:

```python
assert not result.acted_before_asking(r"--allow-unrelated-histories")
```

### Skills that call out to services

`fake` puts a stub binary first on `PATH`, backed by a fixture. skillcheck ships a
`gh` fake; the stub answers what the fixture describes and **refuses everything
else loudly**, so a skill reaching for an unanticipated command fails the test
rather than doing something real.

```python
result = run_skill("Archive it", fake={"gh": {"me/proj": {"visibility": "PRIVATE"}}})

assert not result.refusals("gh")     # nothing unexpected was attempted
```

Four things stand between a test and real infrastructure, and the run only starts
once all of them hold:

1. The stub goes first on `PATH`, and skillcheck runs `gh --skillcheck-fake` itself.
2. The agent then runs the same check in a throwaway session of its own. Checking
   skillcheck's environment proves nothing about the shell the agent's tools run
   in, which can rebuild `PATH` from a profile.
3. `GIT_CONFIG_GLOBAL` rewrites every github.com and gitlab.com URL to a path
   that does not exist, so git cannot reach a forge either. It also carries an
   identity, because without one commits fail and the agent starts improvising
   with your real name and email.
4. Anything the fake does not recognise exits non-zero.

Nothing patches Python's `subprocess`: the agent spawns tools from its own shell
in its own process, where Python-level mocking cannot reach.

### Judging

Some things no assertion catches — whether the skill explained itself, whether
it asked the right question. Write a rubric:

```python
verdict = judge("The reply states the code and hedges nothing", result)
assert verdict, verdict.reasoning
```

Write rubrics as requirements, including any exception the skill itself allows: a
rubric that overstates a rule fails compliant work.

## Results

Each run records beside the test file that produced it, so a skill carries the
record of what it was tested on:

```
skills/git-graveyard/
├── SKILL.md
├── test.py
└── results.json
```

That makes results reviewable. Someone who runs your skill against a harness you
do not have can send the result back as a pull request, and a bug report can
point at the row that failed.

```sh
skillcheck status skills/
```

```
         claude                     codex
SKILL    STATUS  LAST TESTED  TIME  STATUS  LAST TESTED  TIME
-------  ------  -----------  ----  ------  -----------  ----
justfile pass    today        53s   pass    2 days ago   115s
```

A failure is not automatically a regression: an overloaded API, a timeout, or a
model taking a different-but-valid route all show up as red. Re-run the one test
before you change the skill.

## Commands

```sh
skillcheck test skills/my-skill     # run a skill's tests
skillcheck status skills/           # what was tested, and when
skillcheck doctor                   # installed harnesses and settings
skillcheck new-test skills/my-skill # scaffold test.py
```

## Cost, and what CI can cover

Every test is at least one live model call. Run the tests for the skill you
changed, not the whole tree. skillcheck's own CI covers parsing and the fake-binary
machinery against recorded CLI output; it cannot run real agents, so live
behaviour is checked by hand. Agent CLIs change their flags without notice, and
that is where breakage comes from.

## Adding a harness

Subclass `Harness`, implement `command`, `parse`, and ideally `resume_command`
and `session_id`, then register it in `HARNESSES`. Pull requests welcome.

## Licence

MIT.
