Metadata-Version: 2.4
Name: reproassert
Version: 0.2.3
Summary: Turn GitHub issues into repeatable, sandbox-verified failing pytest tests.
Author: Atomics Hub
License-Expression: MIT
Project-URL: Homepage, https://github.com/Atomics-hub/reproassert
Project-URL: Issues, https://github.com/Atomics-hub/reproassert/issues
Project-URL: Documentation, https://atomics-hub.github.io/reproassert/
Keywords: pytest,github,bug-reproduction,testing,developer-tools
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click<9,>=8.1
Requires-Dist: defusedxml<1,>=0.7
Requires-Dist: rich<16,>=13.9
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: jsonschema<5,>=4.23; extra == "dev"
Requires-Dist: mypy<2,>=1.15; extra == "dev"
Requires-Dist: pytest<10,>=9.0; extra == "dev"
Requires-Dist: pytest-cov<8,>=6.0; extra == "dev"
Requires-Dist: ruff<1,>=0.11; extra == "dev"
Requires-Dist: setuptools==83.0.0; extra == "dev"
Requires-Dist: types-defusedxml<1,>=0.7; extra == "dev"
Dynamic: license-file

# ReproAssert

> The test before the fix.

[![CI](https://github.com/Atomics-hub/reproassert/actions/workflows/ci.yml/badge.svg)](https://github.com/Atomics-hub/reproassert/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-111827.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB.svg)](pyproject.toml)

ReproAssert turns a public GitHub issue into a candidate pytest regression test, then proves that
the test collects and fails consistently on the exact buggy commit inside a locked-down Docker
sandbox. It produces the test patch, a machine-readable evidence report, and one-command replay
before anyone tries to fix the bug.

## See it work

You need Python 3.10+, [uv](https://docs.astral.sh/uv/), and a running Docker Engine or Docker
Desktop. This command uses ReproAssert's public issue #1 and pinned buggy commit. It needs no API
key, makes no model call, and builds the hash-locked verifier image automatically on first use.

```console
uvx reproassert demo
```

The proof ends with artifacts you can inspect and replay:

```text
claim    repeatable_base_failure
outcome  repeatable_base_failure
patch    .../candidate.patch
report   .../reproassert-report.json
replay   reproassert replay .../reproassert-report.json
```

```text
GitHub issue + exact commit
           |
           v
candidate.patch + one-command replay + reproassert-report.json
```

It never edits production code. It never silently falls back to running repository code on your
host. Its strongest public CLI claim is deliberately narrow: **this test produced the same
issue-marked failure on the pinned base revision across repeated sandboxed runs.**

## Install and use

You need Python 3.10+, [uv](https://docs.astral.sh/uv/), and Docker Engine or Docker Desktop.

```console
uv tool install reproassert
```

For a current issue, `HEAD` is resolved and recorded as an exact 40-character SHA. Provider use is
always explicit:

```console
export OPENAI_API_KEY="..."
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 --provider openai
```

For a historical issue, pass the known buggy revision yourself; ReproAssert does not guess history:

```console
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --provider openai
```

That result means the candidate collected and produced a stable, expected failure on the exact base
SHA. It does **not** mean the test passes on a fix, captures the issue's true semantics, or has been
accepted by a maintainer.

### Without uv

```console
python3 -m venv .venv
. .venv/bin/activate
python -m pip install reproassert
reproassert demo
```

The controller supports macOS and Linux. WSL is treated as Linux but is not yet independently
verified. Native Windows execution and Windows containers are unsupported. There is no native
execution fallback.

## Choose how the test is created

Every run requires exactly one candidate source.

### A trusted generator adapter

```console
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --generator-command ./your-trusted-adapter \
  --pass-env PROVIDER_API_KEY
```

The adapter is a program you trust. ReproAssert sends it a bounded JSON request on stdin and expects
one JSON object on stdout containing `test_content`, `expected_symptom`, and `rationale`. It receives
untrusted issue and repository text, so it must keep those inputs in the data plane—not interpret
them as commands. Only environment variables named with `--pass-env` are forwarded.

See the working offline [deterministic adapter](examples/deterministic_generator.py) and the
[architecture](docs/architecture.md) for the protocol and trust boundary.

<details>
<summary>Generator protocol response</summary>

```json
{
  "test_content": "def test_issue_123_reproduction():\n    assert observed == expected, 'duplicate separators remain'\n",
  "expected_symptom": "duplicate separators remain",
  "rationale": "Exercises the user-visible normalization invariant."
}
```

The adapter must emit exactly those three string fields. The expected symptom must appear literally
in the test, normally as its assertion message. Output is capped at 64 KiB and execution at 300
seconds. The command runs directly without shell expansion.

</details>

### A human-written test

For issue `123`, create one synchronous test named `test_issue_123_reproduction`:

```python
from your_package import normalize


def test_issue_123_reproduction() -> None:
    observed = normalize("Alpha  Beta")
    assert observed == "alpha-beta", "duplicate separators remain"
```

Then verify it through the same policy and Docker boundary:

```console
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --candidate-file ./candidate.py \
  --expected-symptom "duplicate separators remain" \
  --rationale "Exercises adjacent-space normalization through the public function."
```

### The built-in OpenAI adapter (explicit opt-in)

```console
export OPENAI_API_KEY="..."
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --provider openai
```

The provider is never selected merely because an API key exists. The default model is
`gpt-5.4-mini`; use `--model MODEL` to choose another. This sends bounded public issue and selected
source context to `https://api.openai.com/v1/responses` and may incur charges on your account.
ReproAssert requests `store: false`, makes no automatic retries, and caps request, response, and
output sizes. Review your provider's data policy before use; the source-context filter is not a
data-loss-prevention system.

## What ReproAssert verifies

The first product profile is intentionally Python + pytest:

| Stage | Required evidence |
| --- | --- |
| Source | Canonical public issue, exact 40-character commit SHA, and files that reconstruct the commit's Git root tree |
| Candidate | One new synchronous pytest test, no production edits, strict static policy, at most 32 KiB |
| Collection | The candidate collects successfully inside Docker |
| Execution | It fails 2-10 times (default 3) with the expected symptom and a stable failure fingerprint |
| Boundary | No network, read-only root and workspace, non-root user, dropped capabilities, bounded CPU, memory, PIDs, time, and output |
| Result | A test-only patch, replay command, and machine-readable evidence report |

Syntax errors, collection/import/setup failures, missing dependencies, generic crashes, timeouts,
unrelated failures, and inconsistent one-off failures are rejected rather than counted as
reproductions. Ordinary `issue` runs do not install project dependencies; the current wedge is best
suited to repositories whose test environment is already self-contained under the strict profile.

The claim ladder stays explicit:

```text
rejected -> collected -> repeatable_base_failure  [public CLI ceiling]
                              |
                              +-> differential_reproduction  [capability-gated evaluation]
                              +-> maintainer_validated        [external evidence only]
```

## Inspect and replay the evidence

Run artifacts live under `$XDG_STATE_HOME/reproassert/runs` or
`~/.local/state/reproassert/runs`. Choose another controller-owned directory with `--run-base`.

- `candidate.patch` adds only `tests/reproassert/test_issue_NUMBER.py`.
- `reproassert-report.json` records issue and source provenance, the candidate and its digest,
  Docker policy and immutable image ID, collection and rerun outcomes, bounded logs, failure
  fingerprint, artifact hashes, and explicit limitations.

Replay a report with controller-owned commands:

```console
reproassert replay ~/.local/state/reproassert/runs/issue-.../reproassert-report.json
```

Replay reacquires and verifies the exact source, regenerates safe pytest arguments, and creates a
new report. It does not execute command-looking fields from the original report. A successful replay
is fresh bounded evidence—not semantic proof.

Print the exact report schema bundled with the installed controller without a network request:

```console
reproassert schema
```

The published schema is also available at
[`reproassert-report.schema.json`](https://atomics-hub.github.io/reproassert/reproassert-report.schema.json).

## Security model

Repository code, issue text, source files, dependencies, generated tests, pytest results, and
imported reports are untrusted. The verifier receives no host secrets, SSH agent, browser state,
cloud credentials, proxy variables, Docker socket, or unrelated host directories. Archive paths and
types are checked, accepted source files must reconstruct the pinned Git tree, and repository code
runs only inside Docker with the recorded restrictions.

Residual risk remains. Docker shares the Linux host kernel or Docker Desktop VM; hostile pytest
code may try to forge in-process result detail; and a user-selected generator adapter is a trusted
host process. Treat `repeatable_base_failure` as bounded evidence, not proof.

Before running an unfamiliar repository, read the [security model](docs/security-model.md),
[threat model](docs/threat-model.md), and [sandbox profiles](docs/sandbox-profiles.md). Report
vulnerabilities through GitHub's private process described in [SECURITY.md](SECURITY.md), not a
public issue.

## Benchmark status

Measured results remain intentionally separate from product capability.

- v0.1 is immutable at **0/20** because of its [provenance erratum](benchmarks/v0.1/ERRATA.md).
- v0.2 freezes 20 leak-audited cases from pinned upstream data and independently attests the parser
  boundary and exact Git object graph.
- The v0.2.1 campaign evaluated the complete **20/20 denominator** and accepted **0/20** as L1
  deterministic reproductions. Seventeen outputs failed the frozen candidate contract; three
  reached six-run Docker evaluation but their JUnit files were lost by an evaluator transport bug,
  so they correctly failed closed without an attributable fingerprint. The v0.2.1 result remains
  immutable and is not retroactively upgraded.
- The frozen OpenAI run made exactly 20 calls for **$0.688111 total**, with a maximum case cost of
  **$0.051351**, under the approved $5 total / $0.25 per-case zero-overage limits. Cost per success
  is undefined because there were no successes.
- The result has **0 L2 claims, 0 human reviews, 0 maintainer validations, and no outreach**. It
  misses the preregistered 6/20 continuation gate and is evidence against claiming benchmark-ready
  accuracy for the current generation profile.

The v0.2 preparation and evaluation machinery is default-deny: source, dependency, hidden-fix,
request, pricing, authorization, causal-control, and reviewer commitments must be bound before a
provider-capable run. Public issue creation now precedes the fixing artifact for 20/20 cases under a
hash-bound chronology receipt. The dataset title/body snapshots remain
`historical_public_contamination_exposed` because full pre-fix revision capture is unavailable; no
historical-cleanliness claim is implied.

The executed v0.2.1 path was fail-closed on spend: it reserved each case before inference, persisted
provider response cost before completion, disallowed overage, and resumed only from hash-bound
durable receipts. The public result bundle contains the aggregate and redacted per-case evidence;
provider responses, hidden fixes, credentials, and private evaluator paths are not published.

See the [v0.2 protocol](benchmarks/v0.2-draft/README.md), [evaluation model](docs/evaluation.md), and
[market-validation gates](docs/market-validation.md). The failed 6/20 continuation gate means the
next work is generator/contract diagnosis, not a larger paid run or a performance claim.

## Development

```console
git clone https://github.com/Atomics-hub/reproassert.git
cd reproassert
uv sync
uv run pytest
uv run ruff check .
uv run mypy src
```

Start with [CONTRIBUTING.md](CONTRIBUTING.md). The deeper product contract lives in:

- [Architecture](docs/architecture.md)
- [Security model](docs/security-model.md)
- [Evaluation protocol](docs/evaluation.md)
- [Roadmap](docs/roadmap.md)
- [Business model](docs/business-model.md)
- [Launch plan](docs/launch-plan.md)

ReproAssert is alpha software available under the [MIT License](LICENSE).
