Metadata-Version: 2.4
Name: triagekit-mcp-flaky-test-detective
Version: 0.1.0
Summary: MCP server that tells flaky tests from genuinely broken ones using real CI run history, and generates synthetic test data / API contract diffs for QA automation.
Project-URL: Homepage, https://github.com/Karthick-dev-cart/triagekit-mcp
Project-URL: Repository, https://github.com/Karthick-dev-cart/triagekit-mcp
Author: TriageKit Contributors
License-Expression: MIT
Keywords: ci,flaky-tests,mcp,model-context-protocol,qa,testing
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.11
Requires-Dist: faker>=30.0
Requires-Dist: fastmcp<4.0,>=3.0
Requires-Dist: httpx>=0.27
Requires-Dist: triagekit-mcp-core<0.2.0,>=0.1.0
Description-Content-Type: text/markdown

# triagekit-mcp-flaky-test-detective

Part of the [TriageKit](../../README.md) MCP server suite.

Tells flaky tests from genuinely broken ones using real CI run history — instead of an
engineer guessing "probably flaky, re-run it" or a broken test getting rubber-stamped
past review because nobody has time to check.

## Why

CI red gets treated one of two bad ways: blindly re-run until green (masking a real bug),
or dismissed as "that test is always flaky" (hiding a regression). This server pulls a
test's actual pass/fail history from GitHub Actions, classifies the pattern
deterministically, clusters failures by root cause, and checks for infra correlation —
so the decision is based on evidence, not vibes.

## Tools

| Tool | What it does |
|---|---|
| `classify_test_failure` | Classifies a test's recent history as consistently-failing / consistently-passing / intermittent (flaky) / insufficient-data, with run-by-run evidence |
| `get_flaky_test_leaderboard` | Ranks all tests in a repo's recent CI history by computed flake rate |
| `diff_failure_stack_traces` | Clusters a test's failures by stack-trace similarity — is "flaky" really 2-3 distinct root causes? |
| `correlate_flake_with_infra` | Checks whether flake rate correlates with the CI runner — infra contention vs. genuine non-determinism |
| `suggest_quarantine_candidates` | Returns genuinely intermittent tests above a flake-rate threshold |
| `file_flaky_test_ticket` | Opens a Jira ticket pre-filled with flake evidence (optional — requires Jira config) |
| `generate_synthetic_test_data` | Generates synthetic fixtures from a JSON-Schema fragment or example payload |
| `diff_api_contract` | Diffs two OpenAPI specs and classifies each change as breaking or non-breaking |

## Install

> **Not yet published.** This package is not on PyPI or GHCR yet (alpha, pre-release) - the
> commands below are the intended install path once `triagekit-mcp-core` and this package are
> published (core must publish first; see [CONTRIBUTING.md](../../CONTRIBUTING.md#core-first-release-order)).
> Until then, install from a local build: `uv build --package triagekit-mcp-core --out-dir dist &&
> uv build --package triagekit-mcp-flaky-test-detective --out-dir dist && uv pip install dist/*.whl`.

```bash
uvx triagekit-mcp-flaky-test-detective
# or
pip install triagekit-mcp-flaky-test-detective
# or
docker run --rm -i -e TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN ghcr.io/Karthick-dev-cart/triagekit-flaky-test-detective
```

## Quick Start (Claude Desktop / Claude Code)

Add to your `mcp.json`:

```json
{
  "mcpServers": {
    "flaky-test-detective": {
      "command": "uvx",
      "args": ["triagekit-mcp-flaky-test-detective"],
      "env": {
        "TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN": "ghp_..."
      }
    }
  }
}
```

## Example prompts

- "Is `tests/test_checkout.py::test_apply_discount` actually flaky, or just broken? Check the last 30 runs on `myorg/myrepo`."
- "Give me the flakiest tests in `myorg/myrepo` over the last 20 CI runs."
- "Are `test_upload_large_file`'s failures all the same root cause, or several?"
- "Does `test_db_connection` fail more often on a specific runner?"
- "Which tests in `myorg/myrepo` are safe to quarantine right now?"
- "Generate 20 synthetic user records matching this example: `{\"id\": 1, \"email\": \"a@b.com\", \"active\": true}`."
- "Did this OpenAPI spec change introduce any breaking changes vs. the previous version?"

## Authentication

| Variable | Required for | Notes |
|---|---|---|
| `TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN` (or native `GITHUB_TOKEN`) | All 5 CI-analysis tools | A GitHub PAT (classic or fine-grained) with `actions:read` on the target repo |
| `TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_BASE_URL` | `file_flaky_test_ticket` | e.g. `https://yourteam.atlassian.net` |
| `TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_EMAIL` | `file_flaky_test_ticket` | Jira Cloud account email |
| `TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_API_TOKEN` (or native `JIRA_API_TOKEN`) | `file_flaky_test_ticket` | [Create one here](https://id.atlassian.com/manage-profile/security/api-tokens) |

`generate_synthetic_test_data` and `diff_api_contract` need no credentials — they operate
on data you pass directly. See [`.env.example`](.env.example) for a local-dev template.

Credentials are only ever read from the environment at startup — no tool accepts a token
as a parameter. See [CONTRIBUTING.md](../../CONTRIBUTING.md) for why.

## Development

```bash
uv sync
uv run pytest packages/flaky-test-detective -v
uv run ruff check packages/flaky-test-detective
uv run pyright packages/flaky-test-detective
```

## License

[MIT](../../LICENSE)
