Metadata-Version: 2.5
Name: flake-exorcist
Version: 0.1.0
Summary: Differential-diagnosis flaky test detector for pytest suites
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: pytest
Requires-Dist: pytest-xdist
Requires-Dist: rich
Provides-Extra: dev
Requires-Dist: hypothesis; extra == 'dev'
Requires-Dist: mutmut; extra == 'dev'
Requires-Dist: mypy~=1.11; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff~=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# flake-exorcist

Find flaky tests. Name the cause. Reproduce on demand.

## The problem

Your CI turns red on unchanged code. You re-run the build and it passes. Nobody
learns why, nobody files a fix, and next week it happens again.

Google's engineering research found roughly 1 in 7 tests in large repositories
eventually behaves this way. Most tools today detect flakiness by re-running
failed tests. That tells you *which* test is flaky but not *why*. So you're left
guessing.

## What this does

`flake-exorcist` runs your pytest suite under controlled perturbations, one
variable at a time, and watches which change flips a passing test to failing.
The variable that flips the outcome *is* the root cause.

It needs no CI history, no paid account, and no setup beyond `pip install`.

```
$ exorcist hunt tests/ --fast

Exorcist Report | Runs: 41 | Seed: 20260808 | Flaky: 2 | Env-dependent: 1 | Failing: 1

Flaky Tests
  MEDIUM  tests/test_order_dependent.py::test_counter_is_zero   cause: order   confidence: 57%
    Polluter: tests/test_polluter.py::test_polluter_increments
    Fix: Isolate shared state per test; reset module-level globals in an autouse fixture.
  MEDIUM  tests/test_hash_seed.py::test_unseeded_rng_assumption  cause: seed   confidence: 43%
    Fix: Seed the RNG in a fixture; sort collections before asserting.

Non-portable (not flaky)
  tests/test_timezone.py::test_utc_date_format

Genuinely failing (not flaky)
  tests/test_real_failure.py::test_always_fails
```

Confidence reflects how often the flipping dimension actually reproduced the
failure, so probabilistic flakes (order, parallel) report moderate confidence
rather than a misleading 100%.

Exit code 1 means flaky tests were found. Wire it into CI as a gate.

## Install

Requires Python 3.11 or newer.

```bash
pip install flake-exorcist
```

Or install from source for development:

```bash
git clone https://github.com/suletetes/flake-exorcist.git
cd flake-exorcist
pip install -e ".[dev]"
```

## Quick start

```bash
# Run against the bundled sample repo (ships with the package)
exorcist hunt examples/sample_flaky_repo/tests --fast

# Run against your own project
exorcist hunt path/to/your/tests --fast
```

## The four dimensions

The tool perturbs one thing at a time and observes what happens:

| Dimension | What changes | What it catches |
|---|---|---|
| ORDER | Execution order (seeded shuffle) | Tests leaking shared state to later tests |
| ENVIRONMENT | Timezone, locale | Tests assuming the local clock or locale |
| PARALLEL | Number of workers (pytest-xdist) | Race conditions, port conflicts, file locks |
| SEED | PYTHONHASHSEED and random seed | Tests depending on set/dict ordering or unseeded RNG |

## How the diagnosis works

1. **Baseline** - run the suite once in default order, record every outcome.
2. **Broad scan** - run once per dimension, changing only that variable.
   Tests whose outcome flipped become candidates.
3. **Deep dive** - repeat the flipping dimension multiple times per candidate
   to confirm and score confidence.
4. **Bisection** (for ORDER) - binary search the preceding tests to name the
   exact polluter that contaminates the victim.
5. **Report** - ranked findings with cause, confidence, a reproduction command,
   and a one-line fix hint.

The classifier and attributor are pure functions: same inputs always produce the
same output. They contain no I/O, no clock reads, no subprocess calls. This is
what makes the tool itself testable with property-based methods (see below).

## Verdicts

| Verdict | Meaning |
|---|---|
| `deterministic_pass` | Always passes. Stable. |
| `deterministic_fail` | Always fails. A real bug, not a flake. |
| `flaky` | Outcome changes under a specific dimension. The tool names the cause. |
| `environment_dependent` | Fails under a specific TZ/locale but does so consistently. Non-portable, not flaky. |
| `inconclusive` | Something flipped but evidence is too weak to attribute. Reported honestly. |

## Commands

### `exorcist hunt <path>`

| Flag | What it does |
|---|---|
| `--fast` | Default. One pass per dimension in the broad scan, 3 repeats in deep dive. |
| `--thorough` | 7 repeats per candidate, higher confidence, slower. |
| `--json` | Machine-readable JSON to stdout. Schema documented in design.md. |
| `--only <node_id>` | Skip the broad scan, deep-dive one specific test directly. |
| `--max-runs N` | Hard budget on total suite runs. Partial results on exhaustion. |
| `--dimensions ORDER,SEED` | Test only a subset of dimensions (comma-separated, case-insensitive). |
| `--seed N` | Base seed for reproducibility. Same seed = same run. |
| `--workers N` | Worker count for the parallel dimension. |

### `exorcist repro <victim> --after <polluter>`

Runs exactly two tests in the given order and reports whether the victim fails.
Turns a "random" failure into a deterministic, scriptable one.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | No flaky tests found |
| 1 | Flaky tests detected |
| 2 | Internal tool error |
| 3 | Bad arguments, missing pytest, no tests found |

`environment_dependent` and `deterministic_fail` alone do not cause exit 1.
Only actual flakiness does.

## Configuration

Optional. Put this in your project's `pyproject.toml`:

```toml
[tool.exorcist]
mode = "fast"
dimensions = ["order", "environment", "parallel", "seed"]
base_seed = 42
max_runs = 50
workers = 4
timeout_s = 60
```

CLI flags override config values. Config values override built-in defaults.

## How it was built (Kiro usage)

This project was built with Kiro's spec-driven workflow from start to finish.
The `.kiro/` directory at the root of this repo contains the full trail:

```
.kiro/
  steering/           <- persistent project context applied to every task
    product.md        <- vision, users, non-negotiable principles
    tech.md           <- stack decisions, pytest-driving mechanics
    structure.md      <- repo layout, layering rules, adapter seam
    flakiness-taxonomy.md <- the domain model the classifier encodes
  specs/flake-exorcist/
    requirements.md   <- 67 functional requirements in EARS notation
    design.md         <- architecture, algorithms (pseudocode), JSON schema, 7 ADRs
    tasks.md          <- 27 ordered TDD tasks with requirement traceability
  hooks/
    self-diagnose.json  <- runs exorcist on its own suite after test tasks
```

**The spec-to-code chain:**

1. Described the problem in plain language. Kiro generated EARS-format
   requirements, a design with data models and algorithms, and an ordered task
   list.
2. The classifier and attributor are pure functions (no I/O). Kiro's
   property-based tests verify they are deterministic: same run records in,
   same verdict out, regardless of input order or timestamp noise.
3. A self-diagnose hook runs `exorcist hunt tests/` on this project's own
   suite after test-related tasks. If the tool introduces flakiness into
   itself, it catches it.

The one sentence version: I used Kiro's spec workflow to design the perturbation
engine, and its property-based tests to prove the flakiness classifier is itself
not flaky.

## Testing

```bash
# Full suite
pytest tests/

# Only unit tests (fast, no subprocess)
pytest tests/unit/

# Only property tests (Hypothesis)
pytest tests/property/

# Integration (runs the real CLI against the sample repo)
pytest tests/integration/

# Lint and types
ruff check src/ tests/
mypy --strict src/exorcist
```

The test pyramid:

- **Unit** (`tests/unit/`) tests the pure classifier, attributor, config, models,
  and command-builder in isolation. No subprocesses.
- **Property** (`tests/property/`) uses Hypothesis to assert invariants over
  randomized inputs: classification is deterministic, confidence is monotonic,
  permuting input order changes nothing.
- **Integration** (`tests/integration/`) runs the full CLI against the bundled
  sample repo and asserts correct verdicts, causes, exit codes, and JSON schema.

## Prior art and how this differs

This tool stands on the shoulders of prior work. The approach (perturb hidden
variables to expose flakiness) is not new. What is new is packaging it as a
single local command that crosses all four dimensions on a cold repo without
requiring CI history or a paid account.

| Tool | What it does | How this differs |
|---|---|---|
| [NonDex](https://github.com/TestingResearchIllinois/NonDex) (2016) | Perturbs JVM non-deterministic APIs | Python-native; adds multi-dimension triage |
| [detect-test-pollution](https://github.com/asottile/detect-test-pollution) | Bisects order to find a polluter | Automates discovery (you don't need to know which test is flaky); adds dimensions beyond order |
| [pytest-randomly](https://github.com/pytest-dev/pytest-randomly) | Shuffles test order | We credit the technique; our order-plugin owns order so it can be decoupled from RNG state |
| [Shaker](https://doi.org/10.1145/3395363.3397385) (2020) | Environment noise for concurrency flakiness | Combines env with order, parallel, and seed; classifies rather than just detecting |
| Datadog / Trunk / Mergify | SaaS flakiness dashboards | Local-only, no account needed, works on a repo you cloned ten seconds ago |

## Sample flaky repo

The `examples/sample_flaky_repo/` directory ships a self-contained pytest suite
with one planted flake per dimension plus stable and failing controls:

| Test file | What it demonstrates |
|---|---|
| `test_stable.py` | Always passes (control) |
| `test_real_failure.py` | Always fails (control, not flaky) |
| `test_order_dependent.py` | Fails when run after `test_polluter.py` (shared state) |
| `test_timezone.py` | Fails under `TZ=Pacific/Auckland` (non-portable, not flaky) |
| `test_hash_seed.py` | Fails under some RNG/hash seeds (unseeded randomness) |
| `test_concurrency.py` | Fails under parallel workers (unlocked shared-file race) |

Run it with plain pytest to see the flakiness manifest, or with exorcist to see
it diagnosed.

## Roadmap (out of scope for v1)

- Auto-fixing (suggest or apply patches for common patterns)
- Web dashboard for tracking trends over time
- IDE integration (VS Code, PyCharm inline annotations)
- Adapters for other frameworks (Jest, go test, JUnit)

## License

MIT. See [LICENSE](LICENSE).
