Metadata-Version: 2.5
Name: agent-exam
Version: 0.2.0
Summary: Eval framework for agent skills, across Claude Code, Codex CLI, Copilot CLI and OpenCode
Project-URL: Documentation, https://agent-exam.readthedocs.io
Project-URL: Source, https://github.com/zytedata/agent-exam
Project-URL: Changelog, https://github.com/zytedata/agent-exam/blob/main/CHANGES.rst
Author: Mikhail Korobov
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent,claude-code,codex,eval,llm,skills
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: claude-measure-usage>=0.1.0
Requires-Dist: click>=8.1
Requires-Dist: pydantic>=2.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Description-Content-Type: text/markdown

# agent-exam

[![docs](https://readthedocs.org/projects/agent-exam/badge/?version=latest)](https://agent-exam.readthedocs.io)

Eval framework for agent skills. Point it at a repo of skills and a suite of
YAML tasks, and it runs them against a real agent CLI, grades the resulting
transcript, and reports verdicts plus cost and token metrics.

Supported harnesses: **Claude Code**, **Codex CLI**, **Copilot CLI** and
**OpenCode**. Each runs as a subprocess against your existing subscription or
login — no pay-per-token API key is required.

## Intended use

Interactive development of skills — edit a skill, run the relevant suite or a
single task, read the report, iterate. Automating runs against
subscription-backed CLIs may stretch a harness's terms, and LLM evals are slow
and expensive enough that running every suite on every change buys little extra
signal.

## Install

```bash
pip install agent-exam
```

## Quick start

In the repo holding your skills:

```yaml
# evals/config.yaml
default_harness: claude_code
```

Skills are picked up from `./skills` by default; set `skills_dirs` to point
elsewhere.

```yaml
# evals/suites/my-suite/tasks/greets.yaml
description: |
  The skill should greet by name.
kind: execute
prompt: |
  Greet Ada.
assertions:
  - skill_invoked: my-skill
  - judge: |
      The reply greets Ada by name.
  - no_permission_errors:
```

```bash
agent-exam doctor          # preflight checks
agent-exam my-suite        # run the suite
agent-exam show latest     # read the report
```

`examples/config.yaml` documents every configuration key, and the
[documentation](https://agent-exam.readthedocs.io) covers writing suites,
grading with judges and reading the reports.

The evals directory is found relative to the nearest `pyproject.toml` and
defaults to `evals/`. Add a `[tool.agent-exam]` section only to point somewhere
else, or to register a `pre_run_hook`:

```toml
# pyproject.toml
[tool.agent-exam]
evals_dir = "qa/evals"
pre_run_hook = "evals.hooks:pre_run_hook"
```

## What it gives you

- **Multi-harness by design.** Assertions and the normalized transcript are
  harness-neutral, so adding another agent CLI is a small adapter rather than a
  core rewrite.
- **Trigger evals are first-class.** Test whether the right skill fires for a
  given prompt, separately from whether it then does the right thing.
- **Reality-check modes.** `--without-skill` re-runs a suite with one skill
  removed and `--no-skills` with the whole library removed, so you can see
  whether a skill earns its place against the bare agent.
- **Tags for the costly evals.** Tag a task or a whole suite, declare the tag
  excluded by default, and a run spanning every suite covers the cheap tasks
  only; `--tag` and `--all-tags` bring the rest back.
- **Deterministic and LLM-judge assertions**, per-attempt fixtures and isolated
  working directories, parallel execution with concurrency groups, and
  regression reports across runs (`runs`, `show`, `history`, `diff`).
- **Rescoring without re-running.** `agent-exam rescore` re-grades archived
  attempts against current assertions.

## Contributing

```bash
uv sync
uv run pytest                  # unit tests
tox                            # full matrix, linting, packaging checks
pre-commit install
```

Pull requests run a secrets scan that must pass before merging;
`pre-commit install` runs the same check locally. If it flags a value that is
genuinely not a secret, audit it into `.secrets.baseline` and mention it in the
PR.

## License

Apache-2.0. See [LICENSE](LICENSE).
