Metadata-Version: 2.4
Name: vacuous
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Rust
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
License-File: LICENSE-MIT
License-File: LICENSE-APACHE
Summary: Finds Python tests that pass no matter what your code does
Keywords: testing,linter,pytest,test-quality,static-analysis
License-Expression: MIT OR Apache-2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Issues, https://github.com/MahdiAlani/vacuous/issues
Project-URL: Repository, https://github.com/MahdiAlani/vacuous

# vacuous

Finds Python tests that pass no matter what your code does.

A test with no assertions, or one that only checks its own mocks, still shows up
green and still counts as coverage. Those are the worst kind to have, because
they cost you the thing tests are for — you find out the code is broken from
somewhere other than your test suite.

```console
$ vacuous check tests/

  12 vacuous tests in 847 (1.4%)

  tests/test_billing.py:41   no-assertions              certain
                             └─ `test_refund_is_recorded` contains no assertions — it passes unless the code under test raises.
  tests/test_users.py:118    swallowed-failure          certain
                             └─ this assertion in `test_email_is_unique` is caught and discarded by the handler on line 121 — it can never fail the test.
  tests/test_orders.py:92    patched-target-under-test  likely
                             └─ `test_charge_card` replaces `charge_card` with a mock and then only asserts on that mock — the real `charge_card` never runs.

  63 files scanned
```

It's a static analysis pass over tree-sitter, so it doesn't run your tests and
doesn't need your dependencies installed. Whole repos take well under a second.
No network, no API keys.

## Install

```console
cargo install vacuous
```

Or from source, if you'd rather:

```console
git clone https://github.com/MahdiAlani/vacuous
cd vacuous
cargo build --release
```

Needs Rust 1.88 or newer. Wheels are on the way, so `uv tool install vacuous`
will work without a Rust toolchain.

## Usage

```
vacuous check [PATH]              # defaults to the current directory
  --min-confidence <LEVEL>        # certain | likely | possible  (default: likely)
```

Exits `0` when clean, `1` when it finds something, `2` if the tool itself fell
over — so CI can tell a real failure apart from a broken run.

## Checks

| Check | What it means |
|---|---|
| `no-assertions` | Nothing in the test body can fail |
| `constant-assertion` | Every assertion is on literals: `assert True`, `assertEqual(3, 3)` |
| `swallowed-failure` | An `except` block discards the assertion |
| `unreachable-assertion` | The assertion sits after a `return` or `raise` |
| `patched-target-under-test` | The test mocks the function it's named for, then only checks the mock |

Each finding carries a confidence. `certain` means it's a structural fact about
the code; `likely` means there's a judgement call involved. Only `certain` and
`likely` show by default.

## What it deliberately ignores

Tests without assertions aren't automatically bad, which is the main reason
[pytest hasn't shipped a flag for this](https://github.com/pytest-dev/pytest/issues/2706).
Getting the exceptions right is most of the work here, so `vacuous` stays quiet on:

- Tests that delegate to a helper that asserts, including one defined elsewhere
  in the file and reached indirectly. Flask's `common_object_test(app)` is the
  usual shape.
- Assertion helpers that don't look like assertions — `eq_`, `is_`, `ne_` and the
  rest of the nose/SQLAlchemy family.
- Anything where a decorator does the asserting, like SQLAlchemy's
  `@profiling.function_call_count()` or a `@pytest.mark.benchmark` handing off to
  the benchmark fixture. Plain `pytest.mark` labels don't count.
- Tests returning a value, which means something else is driving them.
- Functions nested inside tests. Flask names route handlers `test` and Click
  commands `testcmd`; no runner collects those.
- `assert False, "shouldn't get here"`. That's constant but always *fails*, so
  it's a deliberate marker, not a vacuous test. Ruff's
  [PT015](https://docs.astral.sh/ruff/rules/pytest-assert-always-false/) and
  [B011](https://docs.astral.sh/ruff/rules/assert-false/) cover that case from
  the other direction.
- `except ValueError: pass` around an assertion. An `AssertionError` escapes it,
  so nothing is being swallowed.

Most of those came from running it over real suites and finding out it was wrong.
It's currently checked against flask, requests, httpx, rich, click, black,
scrapy, celery, pydantic, ansible, django-rest-framework and sqlalchemy — about
29,000 tests — with every finding read by hand. Roughly 2% of tests in those
projects can't fail.

If you hit a false positive, that's a bug worth filing.

## How it works

Walk for files a test runner would collect, respecting `.gitignore`. Parse each
one, pull out the test functions, run every check over them, sort by location.
Files are handled in parallel.

The Python-specific parts live behind one trait in
[`src/lang/mod.rs`](src/lang/mod.rs), so the checks themselves don't know what
language they're looking at. A check is a function from one parsed test to a list
of findings; they're in [`src/rules/`](src/rules/). Every check has a pair of
fixtures under `tests/fixtures/`, one that should be flagged and one that
shouldn't.

## Related tools

- [mutmut](https://github.com/boxed/mutmut) and
  [cosmic-ray](https://github.com/sixty-north/cosmic-ray) answer a harder version
  of this question by mutating your code and re-running the suite. Far more
  thorough, and much slower. Worth reaching for when you want a real answer about
  a specific module.
- [flake8-aaa](https://github.com/jamescooke/flake8-aaa) lints tests for
  Arrange-Act-Assert structure, which catches some of the same shapes.
- [ruff](https://docs.astral.sh/ruff/) has a
  [pytest ruleset](https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt)
  worth turning on regardless, and covers always-false assertions.
- [flake8-pytest-style](https://github.com/m-burst/flake8-pytest-style) for
  pytest idioms more broadly.

`vacuous` is meant to sit alongside these, not replace them.

## Planned

Config in `pyproject.toml`. A baseline file, so an existing project can fail only
on new findings instead of drowning in old ones. JSON and SARIF output. A
pre-commit hook. Wheels, so `uvx vacuous` works.

After that, a `verify` mode: stub out a function, run the tests, and see whether
anything notices. Scoped to a diff it's quick enough to be useful, and it answers
the question this tool can only approximate statically.

TypeScript, eventually.

## License

MIT or Apache-2.0, at your option.

