Metadata-Version: 2.4
Name: ruinator
Version: 0.1.0
Summary: LLM-augmented mutation-testing framework: inject realistic bugs, see if your tests catch them.
Project-URL: Homepage, https://github.com/nauski/ruinator
Project-URL: Repository, https://github.com/nauski/ruinator
Project-URL: Issues, https://github.com/nauski/ruinator/issues
Author: Juha Kesti
License: Apache-2.0
License-File: LICENSE
Keywords: ci,llm,mutation-testing,quality,tdd,testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.40
Requires-Dist: coverage>=7.6
Requires-Dist: jinja2>=3.1
Requires-Dist: litellm>=1.50
Requires-Dist: pathspec>=0.12
Requires-Dist: pydantic>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: tree-sitter-javascript>=0.23
Requires-Dist: tree-sitter-python>=0.23
Requires-Dist: tree-sitter>=0.23
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# Ruinator

> LLM-augmented mutation-testing framework. Inject realistic bugs into your code, see if your tests catch them.

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![Status: alpha](https://img.shields.io/badge/status-alpha-orange)](#status)

Ruinator is a **mutation-testing framework** for people who want to know whether their test suite actually catches bugs. It analyzes your source code, introduces realistic bugs (logic errors, off-by-one mistakes, missing guards), and reruns your tests against each mutation. Surviving mutants reveal gaps in your coverage.

Unlike classical rule-based mutation testers (Stryker, PIT, Mutmut), ruinator is **plugin-extensible**, **LLM-augmented**, **reproducible via seeds + manifests**, and **CI-native**.

## Status

**Alpha (v0.1).** APIs will change. v0.1 ships rule-based mutation with a heuristic equivalence filter; the LLM equivalence judge and creative-mutant slot are scaffolded but not wired into the default pipeline yet.

## Quick look

```toml
# ruinator.toml
[ruinator]
test_command = "pytest -q"
timeout_seconds = 60
seed = 1
survivor_reruns = 3

[ruinator.scope]
include = ["src/**/*.py"]
exclude = []
# coverage_file = ".coverage"   # optional; when set, only covered lines mutate

[ruinator.selection]
categories = ["logic", "boundary"]
languages = ["python"]
max_mutants = 100
per_file_per_category = 4

[ruinator.llm]
provider = "none"  # "claude" and "litellm" also supported (unused in v0.1 pipeline)
```

```bash
ruinator init                               # scaffold ruinator.toml
ruinator run --config ruinator.toml         # run a mutation campaign
ruinator run --report html --report json    # emit reports alongside the manifest
ruinator run --report markdown --report sarif  # PR-comment + code-scanning outputs
ruinator report .ruinator/runs/latest.json  # re-render reports from a manifest
ruinator replay .ruinator/runs/latest.json  # rerun the same mutants at HEAD
```

### GitHub Action

```yaml
# .github/workflows/ruinator.yml
name: Mutation testing
on:
  pull_request:

permissions:
  contents: read
  pull-requests: write     # for the PR comment
  security-events: write   # for SARIF upload to Code Scanning

jobs:
  ruinator:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nauski/ruinator@v0.1.0
        with:
          config: ruinator.toml
          fail-under: '0.80'
```

The action installs ruinator from PyPI, runs it with your config, emits
markdown + SARIF, posts (or updates) a PR comment with the summary, uploads
the SARIF to GitHub Code Scanning, and attaches all reports as workflow
artifacts. See `action.yml` for the full input list.

## Why ruinator?

| | Classical tools (Stryker, PIT) | Mutahunter | LLMorpheus | **Ruinator** |
|---|---|---|---|---|
| Cross-language | per-tool | yes | JS only | **tree-sitter (many)** |
| LLM-augmented | no | yes | yes | **yes (Claude + BYO)** |
| Bug taxonomy | mech operators | none | none | **categorized** |
| Reproducible (seed+manifest) | partial | no | no | **yes** |
| Plugin framework | per-tool | weak | no | **declarative + code** |
| CI baseline-then-diff | no | no | no | **yes (v0.2)** |
| Web UI | Stryker only | no | no | **v0.2** |
| MCP server | no | no | no | **v0.2** |

## How it works

1. **Scope.** Ruinator resolves your `include`/`exclude` globs, parses `# ruinator: skip` annotations, and (optionally) drops uncovered lines using your `coverage.py` / `lcov` report.
2. **Candidates.** tree-sitter parses each file and queries the AST for candidate nodes per mutator category (conditions, comparisons, boundaries).
3. **Sample.** A seeded RNG samples N candidates per category per file, up to your budget.
4. **Mutate.** Built-in mutators (or your plugins) propose a mutation. A creative slot reserved for LLM-proposed bugs is planned for v0.2.
5. **Apply.** Each mutation lives in its own `git worktree` under `.ruinator/worktrees/<id>/`. Your working tree is never touched.
6. **Test.** Your `test_command` runs in the worktree with a timeout. Exit code 0 = survived; non-zero = killed.
7. **Rerun survivors.** Survivors run N more times (default 3) to rule out flaky-pass.
8. **Report.** JSON manifest (canonical) + HTML report (browser-friendly).

## Differentiators

- **Plugin framework** — declarative YAML mutators (`query + prompt template + category`) or code mutators (Python class). Discovered via entry-points.
- **Bug taxonomy** — every mutation carries a category (`logic`, `boundary`, and in v0.3 `security`, `concurrency`). Budgets allocate per category.
- **Reproducibility** — mutation IDs are `sha256(file, node_range, mutator_id, seed)`. Replay a manifest and get the same mutant set every time.
- **Worktree isolation** — one `git worktree` per mutant. Safe for parallel execution and for security mutants that look like exploit code.
- **Cost control** — token budget enforced hard. Prompt caching + Haiku/Sonnet model tiering cut token spend aggressively.
- **Claude-ready** — Anthropic SDK adapter with prompt caching + model tiering, plus LiteLLM for BYO providers. Pipeline defaults to rule-based mutation in v0.1; LLM judging/creation land in v0.2.

## Roadmap

- **v0.1 (current)** — Python + JS, logic + boundary categories, CLI, JSON/HTML reports, heuristic equivalence filter, worktree isolation, manifest cache.
- **v0.2** — LLM equivalence judge wired into the pipeline, creative-mutant slot (Claude/LiteLLM proposing realistic bugs), Web UI, MCP server, GitHub Action, SARIF + Markdown reporters, local LLM provider.
- **v0.3** — Security + concurrency mutators, multi-file mutations, Go/Rust/TS/Java adapters.
- **v0.4** — Auto-test-writer agent (writes tests that kill survivors).
- **v1.0** — Stable plugin API (semver), docs site, hosted SaaS beta.

See [ROADMAP.md](ROADMAP.md) for details.

## Contributing

PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

Development install:

```bash
git clone https://github.com/nauski/ruinator
cd ruinator
uv sync --extra dev
uv run pytest
uv run ruinator --help
```

## Prior art & inspiration

- [Mutahunter](https://github.com/codeintegrity-ai/mutahunter) — language-agnostic LLM mutation testing; closest existing tool.
- [LLMorpheus](https://github.com/githubnext/llmorpheus) — GitHub Next research, JS-only placeholder approach.
- [Meta ACH](https://engineering.fb.com/2025/09/30/security/llms-are-the-key-to-mutation-testing-and-better-compliance/) — multi-agent mutation + test generation at industrial scale.
- [Stryker](https://stryker-mutator.io/), [PIT](https://pitest.org/), [Mutmut](https://github.com/boxed/mutmut), [Cosmic Ray](https://github.com/sixty-north/cosmic-ray) — classical rule-based tools.
- [honnibal/claude-skills #mutation-testing](https://github.com/honnibal/claude-skills/blob/main/mutation-testing.md.txt) — Matthew Honnibal's manual Claude skill; the conceptual starting point.

## License

Apache 2.0. See [LICENSE](LICENSE).
