Metadata-Version: 2.4
Name: mentistest-coverage
Version: 0.4.0
Summary: Analyze test strategy across SaaS products with coverage metrics
Author-email: Harry Trott <harrydtrott@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Harry Trott
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/HarryDouglas/test_coverage_tool
Project-URL: Repository, https://github.com/HarryDouglas/test_coverage_tool.git
Project-URL: Bug Tracker, https://github.com/HarryDouglas/test_coverage_tool/issues
Keywords: test-coverage,coverage-metrics,saas
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.0
Requires-Dist: fastapi<1,>=0.104
Requires-Dist: uvicorn<1,>=0.24
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: jsonschema<5,>=4.0
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: typer<1,>=0.9
Requires-Dist: anthropic<1,>=0.40
Requires-Dist: httpx<1,>=0.25
Requires-Dist: slowapi<1,>=0.1.9
Requires-Dist: tree-sitter<1,>=0.21
Requires-Dist: tree-sitter-language-pack<2,>=0.2
Requires-Dist: tomli<3,>=2.0; python_version < "3.11"
Provides-Extra: test
Requires-Dist: pytest~=9.0; extra == "test"
Requires-Dist: pytest-cov>=4.1; extra == "test"
Requires-Dist: pytest-asyncio>=0.21; extra == "test"
Requires-Dist: allure-pytest~=2.15; extra == "test"
Requires-Dist: packaging>=21.0; extra == "test"
Requires-Dist: hypothesis<7,>=6.0; extra == "test"
Provides-Extra: coverage
Requires-Dist: coverage>=5.0; extra == "coverage"
Provides-Extra: types
Requires-Dist: types-PyYAML>=6.0; extra == "types"
Provides-Extra: code-checks
Requires-Dist: black==25.1.0; extra == "code-checks"
Requires-Dist: mypy==1.16.1; extra == "code-checks"
Requires-Dist: pydocstyle==6.3.0; extra == "code-checks"
Requires-Dist: pylint==3.3.7; extra == "code-checks"
Requires-Dist: ruff>=0.1.0; extra == "code-checks"
Provides-Extra: docs
Requires-Dist: sphinx~=8.1; extra == "docs"
Requires-Dist: sphinx-breeze-theme; extra == "docs"
Requires-Dist: sphinxcontrib-mermaid; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints; extra == "docs"
Requires-Dist: sphinx-book-theme; extra == "docs"
Requires-Dist: myst-parser; extra == "docs"
Provides-Extra: dev
Requires-Dist: invoke~=2.2; extra == "dev"
Requires-Dist: changelog-cli; extra == "dev"
Requires-Dist: cruft; extra == "dev"
Requires-Dist: ipython; extra == "dev"
Requires-Dist: pre-commit~=4.0; extra == "dev"
Requires-Dist: setuptools; extra == "dev"
Requires-Dist: build; extra == "dev"
Provides-Extra: logging-json
Requires-Dist: python-json-logger<4,>=2.0; extra == "logging-json"
Provides-Extra: metrics
Requires-Dist: prometheus-client<1,>=0.20; extra == "metrics"
Provides-Extra: all
Requires-Dist: invoke~=2.2; extra == "all"
Requires-Dist: setuptools; extra == "all"
Requires-Dist: build; extra == "all"
Dynamic: license-file

# mentistest-coverage

[![PyPI Downloads](https://static.pepy.tech/personalized-badge/mentistest-coverage?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/mentistest-coverage)

Strategic test-suite analyser. Looks at a Python project's source tree, test
tree, JUnit XML and (optionally) an OpenAPI spec, and produces a *test-strategy*
report — not just a coverage percentage.

What it does that `coverage.py` doesn't:

- Classifies every test as **unit / integration / contract / e2e** using markers,
  imports, fixtures and directory layout.
- Weights coverage gaps by **cyclomatic complexity and call-graph depth** so the
  20 functions that matter rank above the 200 that don't.
- Correlates tests against the **OpenAPI spec** to flag uncovered endpoints.
- Tracks results over time in a local SQLite **history store** with full git/CI
  provenance, and renders a **trend dashboard** per project.
- Detects **regressions** between snapshots and can fail CI when a previously-
  covered symbol loses coverage.
- Ships an opt-in **AI second opinion** (Claude Haiku → Sonnet escalation, gated
  to skip healthy projects — typical cost <£0.01/run).
- Exports HTML (interactive), JSON (machine-readable) and PDF (board-ready) reports.
- Ships a first-class **GitHub Action** that posts a coverage-delta PR comment
  and uploads artifacts.

Local-first. The analyser runs against filesystem paths; source code is never
uploaded anywhere. The optional AI call sends a structured summary (file paths,
coverage numbers, test names) — never the source itself.

## Install

```bash
pip install mentistest-coverage
```

…or from a clone:

```bash
pip install -e .
```

Verify:

```bash
mentistest --help
```

Every PyPI release is built via OIDC trusted publishing and signed
with Sigstore — see [RELEASING.md](RELEASING.md) for the full chain
and verification recipe.

## Quick start

A complete analysis with all the features wired in:

```bash
mentistest analyze my-service \
    --source ./src --tests ./tests \
    --junit ./reports/junit.xml \
    --api-spec ./openapi.yaml \
    --output-json ./report.json \
    --output-html ./report.html \
    --output-pdf  ./report.pdf \
    --track --compare-baseline --fail-on-regression \
    --detect-git \
    --ai
```

What that does:

| Flag                        | Effect |
| --------------------------- | ------ |
| `--config / -c`             | Project manifest (`mentistest.yaml`); auto-detected when omitted |
| `--source / -s`             | Source directory (repeatable) |
| `--tests / -t`              | Test directory (repeatable) |
| `--junit`                   | JUnit XML report path (repeatable) |
| `--api-spec`                | OpenAPI / Swagger spec — drives endpoint-coverage analysis |
| `--coverage-xml`            | Real coverage.xml from coverage.py (preferred over the fallback) |
| `--coverage`                | Real coverage report in any format (Cobertura, JaCoCo, Istanbul, SimpleCov, LCOV, Go) — auto-detected |
| `--coverage-format`         | Format of `--coverage` (`auto` default, or an explicit format) |
| `--per-test-coverage`       | Per-test coverage (Jest/Vitest Istanbul JSON map or a dir of Go labelled coverprofiles) — populates "tests touching this file" for JS/TS and Go |
| `--per-test-format`         | Format of `--per-test-coverage` (`auto` default, `jest`, `vitest`, `go`) |
| `--output-json / -j`        | JSON report path |
| `--output-html / -H`        | Interactive HTML dashboard path |
| `--output-pdf  / -P`        | Board-ready PDF report path |
| `--track / --no-track`      | Save the run as a snapshot in the history DB |
| `--db-path`                 | Override `~/.test-coverage/history.db` |
| `--compare-baseline`        | Diff this run against the most recent snapshot |
| `--regression-threshold N`  | Allowable coverage drop in pp (default 0) |
| `--fail-on-regression`      | `exit 1` if a regression is detected |
| `--detect-git / --no-detect-git` | Capture commit, branch, CI run ID, provider |
| `--ai`                      | Run the optional AI commentary (requires `ANTHROPIC_API_KEY`) |
| `--test-type`               | Default type label when no signal fires (`unit`/`integration`/`contract`/`e2e`) |
| `--verbose / -v`            | Debug logging |

### Multi-repo projects: the manifest

When different *types* of test live in different *places* — unit tests
in-repo, an e2e suite in a sibling repo, a Pact contract suite elsewhere —
declare them once in a checked-in `mentistest.yaml` and run `mentistest
analyze` with no other flags:

```yaml
project: my-service
source: [src]
tests: [tests/unit, tests/integration]
external:
  - path: ../e2e-suite
    type: e2e
    attribution: endpoint        # match URL literals against the OpenAPI spec
    junit: [../e2e-suite/reports/junit.xml]
  - path: ../contract-tests
    type: contract
    attribution: none
```

CLI flags take precedence over manifest scalars; test locations are
merged. Paths resolve relative to the manifest. See
[the manifest guide](docs/usage.md#project-manifest) for the full schema.

## Server mode

```bash
mentistest serve --host 0.0.0.0 --port 8000
```

Because the `/analyze` endpoint reads arbitrary filesystem paths, `serve`
**refuses to bind a non-loopback host** (anything other than `127.0.0.1`/
`::1`/`localhost`) unless `TEST_COVERAGE_API_KEY` or
`TEST_COVERAGE_ALLOWED_PATHS` is configured. Pass `--insecure` to override
on a trusted private network. The published Docker image binds `0.0.0.0` and
sets `TEST_COVERAGE_REQUIRE_SECURITY=1`, so the container likewise won't
start unprotected.

Then:

- `http://localhost:8000/docs` — interactive OpenAPI docs for the REST API
- `http://localhost:8000/benchmarks` — multi-project benchmark dashboard
- `http://localhost:8000/trends/<project>` — single-project trend dashboard
- `http://localhost:8000/report/html?project_name=<name>` — last-rendered report
- `http://localhost:8000/report/pdf?project_name=<name>` — last-rendered PDF
- `http://localhost:8000/history/<project>` — JSON list of snapshots
- `http://localhost:8000/regression/<project>` — JSON regression diff

## GitHub Action

```yaml
- uses: HarryDouglas/test_coverage_tool@v1
  with:
    source_dirs: src
    test_dirs: tests
    junit_xml_paths: reports/junit.xml
    track_history: 'true'
    fail_on_regression: 'true'
```

Posts a PR comment with the coverage delta, uploads JSON + HTML reports as
workflow artifacts, and fails the check if a symbol loses coverage. See
`action.yml` for all inputs.

**Action vs PyPI wheel:** the action is its own distribution channel —
referenced by git tag, not installed from PyPI. Under the hood it
installs the wheel from PyPI and runs the CLI, so the two move
together but version independently. The action's `@v1` tag pins a
major; minor / patch wheel updates roll through without a workflow
change.

## Security

If you're deploying the HTTP API beyond `localhost`, read
[SECURITY.md](SECURITY.md). Two env vars (`TEST_COVERAGE_API_KEY` and
`TEST_COVERAGE_ALLOWED_PATHS`) enable bearer-token auth and
filesystem path-traversal protection. Both are off by default so local,
loopback-only use is unaffected — but the tool **fails closed** the moment
it would be network-reachable: `serve` rejects a non-loopback `--host`
(unless `--insecure`), and `create_app` refuses to start when
`TEST_COVERAGE_REQUIRE_SECURITY=1` (set in the Docker image) is configured
without a protection layer.

To report a vulnerability, email **harrydtrott@gmail.com** — don't
open a public issue.

## Public benchmark suite

Benchmarking is built around a **frozen corpus** — a small set of checked-in
fixtures (`benchmarks/corpus/`, plus the polyglot monorepo fixture) that ship
their own source and pre-captured coverage / JUnit / OpenAPI artefacts. The
analyser runs over them **offline and deterministically**, so the default
`make benchmark` is fast and never clones, installs, or runs a foreign test
suite. The corpus is the single foundation for three things:

* **Regression gate** — a golden `report.json` per member, diffed in CI on
  every change (`unittests/benchmarks/test_corpus_golden.py`).
* **Accuracy check** — hand-verified ground truth per member
  (`unittests/benchmarks/test_corpus_truth.py`).
* **Showcase dashboard** — the same reports, rendered fast.

```bash
make benchmark         # render the frozen corpus (fast, offline, deterministic)
make benchmark-serve   # serve the dashboard at localhost:8003/benchmarks
make benchmark-check   # fail if any committed report is from an older tool version

make benchmark-refresh # the slow, occasional run against real OSS repos
                       # (clone + install + run their suites) — a real-world
                       # smoke test, not the default
```

Outputs land in `reports/benchmarks/<project>/` (HTML + JSON). After an
*intentional* metric change, regenerate the goldens with
`UPDATE_CORPUS_GOLDENS=1 pytest unittests/benchmarks/test_corpus_golden.py`
and commit them. See `benchmarks/corpus/README.md` for the corpus model and
how to add a member.

## Library use

```python
from pathlib import Path
from test_coverage_tool.core.analyzer import AnalysisEngine
from test_coverage_tool.core.git import detect_git_metadata
from test_coverage_tool.reports.html_generator import HTMLReportGenerator
from test_coverage_tool.reports.json_generator import JSONReportGenerator
from test_coverage_tool.history.store import HistoryStore

engine   = AnalysisEngine()
analysis = engine.analyze(
    project_name     = "my-service",
    source_dirs      = [Path("src")],
    test_dirs        = [Path("tests")],
    junit_xml_paths  = [Path("reports/junit.xml")],
    api_spec_path    = Path("openapi.yaml"),
    git_metadata     = detect_git_metadata(),
)
JSONReportGenerator().write(analysis, Path("report.json"))
HTMLReportGenerator().write(analysis, Path("report.html"))

# Persist a snapshot
HistoryStore(Path("history.db")).save("my-service", analysis_to_dict(analysis))
```

## Configuration

Environment variables:

| Variable                   | Purpose |
| -------------------------- | ------- |
| `ANTHROPIC_API_KEY`        | Enables `--ai` (otherwise the flag is a no-op) |
| `TEST_COVERAGE_DB_PATH`    | Override the default `~/.test-coverage/history.db` location |
| `MENTIS_BENCH_AI`          | Set to `1` to opt-in to AI commentary on `make benchmark` |

## Development

```bash
make install     # install dev deps via uv
make test        # pytest --cov on unittests/
make lint        # ruff + mypy --strict + pylint
make benchmark   # rebuild the public benchmark dashboard
```

The repo uses ruff, mypy (`--strict`), pylint and pytest. The quality bar:
zero ruff/mypy/pylint issues, pytest -x -q must pass before merge.

## License

MIT — see `LICENSE`.
