Metadata-Version: 2.5
Name: pytest-copilot
Version: 2.0.0
Summary: Run GitHub Copilot tasks in pytest. Check results with your own assertions.
Project-URL: Homepage, https://github.com/sbroenne/pytest-copilot
Project-URL: Repository, https://github.com/sbroenne/pytest-copilot
Author: Stefan Brunner
License: MIT
License-File: LICENSE
Keywords: agents,ai,copilot,custom-agents,llm,mcp,pytest,skills,testing
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: github-copilot-sdk>=1.0.28
Requires-Dist: httpx>=0.28.1
Requires-Dist: mcp>=2.3.0
Requires-Dist: pydantic>=2.14.1
Requires-Dist: pytest-asyncio>=1.4.1
Requires-Dist: pytest>=9.2.0
Requires-Dist: python-frontmatter>=1.3.0
Requires-Dist: tomlkit>=0.15.2
Requires-Dist: trio>=0.34.0
Requires-Dist: websockets>=17.3
Provides-Extra: dev
Requires-Dist: pre-commit>=4.7.0; extra == 'dev'
Requires-Dist: pyright>=1.1.414; extra == 'dev'
Requires-Dist: pytest-cov>=7.1.0; extra == 'dev'
Requires-Dist: python-dotenv>=1.2.4; extra == 'dev'
Requires-Dist: ruff>=0.17.3; extra == 'dev'
Requires-Dist: typeguard>=4.6.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: cairosvg>=2.9.1; extra == 'docs'
Requires-Dist: mkdocs-material<9.9,>=9.7.7; extra == 'docs'
Requires-Dist: mkdocs>=1.6.1; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=1.0.8; extra == 'docs'
Requires-Dist: pillow>=13.0.0; extra == 'docs'
Provides-Extra: test
Requires-Dist: syrupy<7,>=6.1.1; extra == 'test'
Description-Content-Type: text/markdown

# pytest-copilot

[![PyPI version](https://img.shields.io/pypi/v/pytest-copilot)](https://pypi.org/project/pytest-copilot/)
[![CI](https://github.com/sbroenne/pytest-copilot/actions/workflows/ci.yml/badge.svg)](https://github.com/sbroenne/pytest-copilot/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Test whether GitHub Copilot can complete tasks with your tools and instructions.**

Independently maintained; not an official GitHub product or endorsed by GitHub.

pytest-copilot runs real Copilot sessions from pytest. Give Copilot a
task, check the actual output with ordinary assertions, and save the execution
as JSON.

Use it to test MCP servers, CLI tools, skills, system prompts, plugins, and custom
agents. It answers a different question from your tool's unit tests: **can
Copilot use the interface correctly?** A working tool can still have confusing
descriptions or arguments that lead the agent to the wrong result.

You define what correctness means. The framework manages the Copilot session,
execution limits, usage, and evidence; it does not grade answers with another AI.

## Quick start

You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a GitHub account with
Copilot access.

Run these commands in an existing Python project with a `pyproject.toml`:

```bash
uv add pytest-copilot
gh auth login --hostname github.com
uv run pytest-copilot init
uv run pytest-copilot doctor
uv run python -m pytest tests/test_copilot_run.py -v
```

`init` creates a test using the bundled Todo MCP server, configures JSON output,
and adds starter-model rates to `pricing.toml`. It reports conflicts rather than
overwriting existing files or settings. `doctor` checks setup and Copilot access.
Real task execution may consume Copilot premium requests.

The test checks the tool's arguments and returned task state. pytest prints the
result; `aitest-reports/results.json` preserves the execution. See the
[setup guide](docs/getting-started/index.md) or the standalone
[quickstart project](examples/quickstart/).

## A test that checks the result

```python
from __future__ import annotations

import json

from pytest_copilot.copilot import CopilotTask


async def test_order_total(copilot_run, tmp_path):
    agent = CopilotTask(
        name="order-total",
        model="gpt-5.6-sol",
        instructions="Save the requested output as JSON. Use integer cents.",
        working_directory=str(tmp_path),
    )
    result = await copilot_run(
        agent,
        "Save order.json for two items at 1250 cents each plus 20% tax. "
        "Include subtotal_cents, tax_cents, and total_cents.",
    )
    assert result.success, result.error
    output = tmp_path / "order.json"
    assert output.is_file()
    assert json.loads(output.read_text(encoding="utf-8")) == {
        "subtotal_cents": 2500,
        "tax_cents": 500,
        "total_cents": 3000,
    }
```

`CopilotTask` configures the session. The second argument to `copilot_run` is
the user task. `result.success` checks execution completion; the file assertions
check that the task produced the right result.

For your own tools, check the resulting application state, saved files, or
operation output. A tool call or a model's claim is not proof of task success.
Use approved isolation for live tests: a temporary directory is not a sandbox,
and generated code or tool calls can act with your user's access.

## What the framework provides

| You need | Use |
| --- | --- |
| Real task execution | `CopilotTask` and the `copilot_run` pytest fixture |
| Controlled runs | Timeouts, tool budgets, request controls, and permission settings |
| Debugging evidence | Configuration, calls, arguments, outputs, errors, and completion records |
| Repeatable comparisons | `ab_run` and pytest repetitions |
| Usage visibility | Captured tokens, Copilot premium requests, and explicit price estimates |
| Saved results | Native JSON with execution evidence and ordinary pytest outcomes |

Your tests supply application fixtures, guarded tool adapters, and independent
output checks. The framework makes one execution attempt per call; it does not
retry failures automatically. Missing usage, prices, and incomplete evidence
remain visible.

## Investigate and compare

Give your coding agent a failed test, its JSON evidence, the requirements, and
the relevant source. It can investigate the failure and help you change the
interface. Keep the success criteria fixed when rerunning a comparison.

For A/B tests, record verification separately for each side: both saved
executions share one pytest outcome. That outcome alone cannot establish
independent pass rates or a causal improvement. See
[comparing configurations](docs/getting-started/comparing.md).

## Documentation

- [Getting started](https://sbroenne.github.io/pytest-copilot/getting-started/)
- [Task guides](docs/how-to/index.md) and [API reference](docs/reference/api.md)
- [Examples](examples/README.md), including historical skill comparisons
- [Upgrading to 2.0](docs/migration.md): new package, imports, and task names,
  without compatibility aliases

No framework companion skill is required or distributed. Testing your own
domain skills is supported.

## License

MIT. Inspired by [agent-benchmark](https://github.com/mykhaliev/agent-benchmark).
