A CLI that diagnoses flaky pytest tests through differential diagnosis. Change one hidden variable at a time, observe which change flips a passing test to failing. That variable is the cause.
pip install flake-exorcist
How It Works
Stop re-running failures and hoping they pass. This tool changes one hidden variable at a time and watches which change flips a green test red. The variable that flipped the outcome is the root cause.
Run the suite once in default order. Record every test outcome. This is the control group that all other runs get compared against.
Re-run the suite once per dimension (order, environment, parallelism, seed). Only one variable changes per run. Tests whose outcome flipped become candidates.
For each candidate, repeat the flipping dimension multiple times and score confidence. If the cause is order-dependence, binary search identifies the exact polluting test.
Ranked findings with cause, confidence percentage, reproduction command, and a fix hint. Exit code 1 gates your CI. JSON output feeds automation.
Four Dimensions
Each dimension isolates one hidden variable. Only one changes per run, so when a test flips, you know exactly what caused it.
Shuffles execution order with different seeds. Catches tests that leak shared state to tests running after them.
Varies timezone and locale. Catches tests that assume the local clock or depend on string collation rules.
Runs serial vs. multi-worker via pytest-xdist. Catches race conditions, port conflicts, and file locks that only surface under concurrency.
Varies PYTHONHASHSEED with order held fixed. Catches tests that depend on set/dict iteration order or unseeded randomness.
Real Output
Run one command against a repo with planted flaky tests. The tool finds the cause, tells you why, and suggests how to fix it.
Five Verdicts
Every test gets one of five labels. The tool says "inconclusive" when it does not have enough evidence, instead of making something up.
Development Process
75 EARS requirements written before any code. 7 architecture decision records. 26 ordered TDD tasks. Property-based tests (Hypothesis) proving the classifier is deterministic. A self-diagnose hook that catches any flakiness the tool introduces into its own suite.
Prior art credited:
NonDex (2016), detect-test-pollution, pytest-randomly, Shaker (2020).
What is different here:
One command, zero CI history needed, works on a cold repo, cross-dimension attribution.