Metadata-Version: 2.5
Name: flakeguard
Version: 0.2.0
Summary: Find, rank, and quarantine flaky tests from any language's test reports.
Project-URL: Homepage, https://github.com/PrathameshKulkarni2k4/flakeguard
Author: Prathamesh Kulkarni
License: MIT
License-File: LICENSE
Keywords: ci,developer-tools,flaky-tests,junit,test-reports,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# flakeguard

### Your test suite is lying to you.

![flakeguard demo](docs/demo.gif)

Flakeguard finds the tests that **randomly pass and fail**, ranks them by how
unreliable they are, and shows you why — so you can trust your tests again.

**Free · private · works with any language.** No account, no sign-up, and
nothing leaves your machine.

[![CI](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml/badge.svg)](https://github.com/PrathameshKulkarni2k4/flakeguard/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)

---

## The problem, in 30 seconds

You have a test that fails once in a while. You re-run it. It passes. Was
that a real bug? A flaky test? Nobody knows — so everyone ignores it, and
that is exactly how real bugs hide.

A flaky test costs a team hours, and slowly destroys trust in the entire
test suite. Flakeguard fixes the hard half of the problem: it tells you
**exactly which tests are unreliable**, and how unreliable.

## Try it in 60 seconds

```bash
pipx install flakeguard
flakeguard scan ./reports        # a folder of test reports
```

No reports yet? Run it against the examples bundled in this repo:

```bash
git clone https://github.com/PrathameshKulkarni2k4/flakeguard
cd flakeguard
flakeguard scan examples/reports
```

## What it does

- Reads your test results from **many runs** (the more runs, the sharper the
  picture).
- Finds the tests that **disagree with themselves** across those runs.
- Gives each one a score from **0 (solid) to 1 (a coin-flip)**.
- Shows the **latest failure message**, so you can go fix it.
- Runs in your **CI** and can leave a comment on the pull request — or block
  it when flakiness spikes.

## Works with any language

Flakeguard reads **JUnit XML**, the report format almost every test
framework can produce. One tool, every language:

| Language / framework | How to produce the report |
|---|---|
| Python (pytest) | `pytest --junitxml=report.xml` |
| Java (Maven / Gradle) | Surefire writes JUnit XML automatically |
| Go | `gotestsum --junitfile report.xml` |
| JavaScript (Jest) | the `jest-junit` reporter |
| Ruby (RSpec) | the `rspec_junit_formatter` gem |
| .NET, PHPUnit, … | built-in or plugin JUnit output |

Flakeguard also reads **TAP** (the Test Anything Protocol), produced by
Node's built-in test runner, the `tap`/`tape` libraries, Perl's Test::More
and others.

## Use in CI (GitHub Actions)

```yaml
- name: Flakeguard
  uses: PrathameshKulkarni2k4/flakeguard@main
  with:
    reports: reports      # dir/glob of JUnit XML files
    min-runs: 3           # ignore tests seen in fewer runs
    max-flaky: 0          # fail if any flaky test is found (-1 = never fail)
    comment: "true"       # post the report as a PR comment
```

See [`.github/workflows/example.yml`](.github/workflows/example.yml) for a
complete workflow that runs a suite several times and feeds the reports in.

To gate locally or in any other CI, use the exit code:

```bash
flakeguard scan ./reports --fail-on-flaky   # exits 1 if anything is flaky
```

## How the score works

For each test we count passes vs. non-passes across all runs (skips are
ignored). The flakiness score is:

```
score = 2 * min(passes, failures) / (passes + failures)
```

- always passes, or always fails → **0.0** (consistent, *not* flaky)
- flips on every run (50/50) → **1.0** (maximally flaky)

It is deliberately explainable — no black box. You can read the number and
know exactly how it was derived.

## Roadmap

- [x] JUnit XML parsing, flakiness scoring, text/JSON/Markdown reports
- [x] GitHub Action with PR comments and a flakiness budget
- [x] TAP report format
- [ ] Native JSON report formats
- [ ] CI history so it learns across builds
- [ ] Failure-signature clustering (group the *same* flake across tests)
- [ ] Quarantine file so known flakes are set aside, not deleted

## Contributing

Contributions are very welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
Adding support for a new report format is a great first contribution and is
designed to be small. Please be kind; this project follows a
[Code of Conduct](CODE_OF_CONDUCT.md).

## License

MIT — see [LICENSE](LICENSE).
