Metadata-Version: 2.5
Name: checkwash
Version: 0.8.0
Summary: Flags known patterns of weakened tests and CI checks in a Git diff.
Project-URL: Homepage, https://github.com/taipei49314/checkwash
Author: taipei49314
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agent,ci,code-review,reward-hacking,test-tampering
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# checkwash

**v0.7.0 preserves bounded imported-helper skip guards, strengthens Git input
failure handling, and detects supported synchronous JS/TS assertions swallowed
by an added catch.**
[Release evidence and known limits](https://github.com/taipei49314/checkwash/blob/v0.7.0/docs/releases/v0.7.0-public-launch.md).
A pass does not prove correctness; retain exit 2 and diagnostics as analysis
errors. The recommended Action is pinned to the same v0.7.0 release.

**Unreleased v0.8.0 candidate (as of this README's commit).** This source is
prepared as v0.8.0. Its Action no longer lets a pull request replace the
installer or checker ([#372](https://github.com/taipei49314/checkwash/pull/372)),
`check --format hook-json` exits 0 with a "not analysed" message on an engine
error, oversized files that no detector reads are skipped with a stderr note,
and commands run from a subdirectory read the repository from its root.
[Candidate evidence and known issues](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.8.0-public-launch.md).
The installation commands below and the recommended Action stay on the last
verified release (CLI and Action v0.7.0) until the v0.8.0 publication
checks pass. A PyPI page keeps the README of its first upload; the
[current README](https://github.com/taipei49314/checkwash#readme) names the
verified release.

**Development snapshots:** the source in this checkout may be newer than the
published CLI. [The advertised-release manifest](https://github.com/taipei49314/checkwash/blob/main/release/advertised.toml)
records the version, commit and download bytes behind the installation commands
below. New changes require their own candidate evidence and remain unreleased
until the publication record verifies them; candidate CI success does not mean
that the recommended Action pin includes them.

[![CI](https://github.com/taipei49314/checkwash/actions/workflows/ci.yml/badge.svg)](https://github.com/taipei49314/checkwash/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/checkwash.svg)](https://pypi.org/project/checkwash/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/taipei49314/checkwash/blob/main/LICENSE)

**Check whether a code change weakens your tests.**

A test can pass after it stops checking something useful. checkwash reads
your Git diff and flags known patterns such as deleted assertions, skipped
tests, relaxed expectations, and CI checks that stop failing.

```diff
- assert total == 105.3
+ assert total > 0
```

The new assertion accepts many incorrect totals. checkwash can flag changes
like this for review, including changes written by coding agents.

**Runs locally. No LLM. No network during analysis. Never executes your code.**

**v0.3.0 replaces five file-wide exemption rules with content-bound ones.**
In v0.2.13 those rules could reuse a recorded approval for a later change to
the same path; v0.3.0 retires their path-only keys. The [upgrade notes](https://github.com/taipei49314/checkwash/blob/main/docs/remediation-upgrade.md)
describe the content-bound replacement, retired keys, and installation fixes.

**v0.4.0 extends detection of runtime suppression, collection changes,
expectation rewrites and substituted subjects, and recognizes more table refactors.** The proofs retain
input identity, execution context and conservative handling of unknown code.
[Release evidence and remaining limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.0-public-launch.md)

**v0.4.1 repairs Node assertion recognition ([#164](https://github.com/taipei49314/checkwash/issues/164))**
and covers Node test-file names, bounded import aliases and lexical shadows.
Coverage warnings expose known assertion candidates the scanner cannot represent;
they do not prove complete JS/TS coverage. The same assertion contract checks
the source, wheel and zipapp. [v0.4.1 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.1-public-launch.md)

**v0.4.2 strengthens JS/TS assertion evidence.** Test callbacks own their
assertions; changing a scalar expected value or loosening positive
`toBeCloseTo` precision reaches the existing detectors. Equivalent scalar
spellings remain equivalent across subject checks. This is bounded syntax
support, not general JS/TS analysis. [v0.4.2 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.2-public-launch.md)

**v0.5.0 closes a round of red-team reports (#172–#181).** A skip in a
fixture or setup the test runs, a selector over edited tests, a package
manifest's test command, Jest's `__tests__/` layout, JS block-level and
imperative skips, JS module mocks, hand-rolled JS tolerances, chai assertions
and CI control flow that stops a runner now reach the detectors; #173, #180
and #181 are closed in part. Some existing findings now carry a different fingerprint,
so allowlist entries for those shapes must be re-recorded; the JSON shape is
unchanged. [v0.5.0 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.5.0-public-launch.md)

**v0.6.0 works through the reports and rulings that followed (#196–#261).**
JS/TS assertions keep their meaning across Jest, chai, Vitest and node:test,
and the runner a file's imports name decides whether a moved test still runs
and what a focus stops. Hand-rolled tolerances in both frontends, pytest
selectors under explicit targets, skip guards in a body, a setup or a conftest
hook, every `pytestmark` spelling and a CI config the reader cannot read now
reach the detectors. Some existing findings now carry a different fingerprint,
so allowlist entries for those shapes must be re-recorded; the JSON shape is
unchanged. [v0.6.0 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.6.0-public-launch.md)

## Try it

You need **Python 3.11+ and Git**. Download and try the offline examples first.

Windows PowerShell (including 5.1):

```powershell
curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.7.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo
```

PowerShell 5.1 aliases `curl` to `Invoke-WebRequest`; use `curl.exe` as written.
macOS/Linux or Git Bash:

```bash
curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.7.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo
```

Then, from the root of a repository with at least two commits:

```bash
python checkwash.pyz check HEAD~1..HEAD
```

This checks your last commit. Start with a change you already understand.
You can also [download the file in your browser](https://github.com/taipei49314/checkwash/releases/download/v0.7.0/checkwash.pyz).
For uncommitted changes use `python checkwash.pyz check`. For a branch review,
use `python checkwash.pyz check BASE...HEAD` to compare from the merge base;
`BASE..HEAD` compares the two named snapshots directly.

Run v0.7.0 from the repository root; from a subdirectory its result is not
reliable. The fix for [#416](https://github.com/taipei49314/checkwash/issues/416)
is on main; it is not released yet. There, `check` from a subdirectory
analyses the whole repository from its Git root and says so in one line on
stderr.

| Result | What to do |
|---|---|
| **Pass · exit 0** | No finding requires blocking under your configuration. Keep running your normal tests and review. |
| **Block · exit 1** | Read the finding and the diff. It may be weakened verification or a false positive. |
| **Error · exit 2** | Resolve the input or analysis error before relying on the result. |

The default threshold is **high**: a visible **warn** can still pass.
`REPAIR_EVIDENCE` describes related changes in the same diff; it does not
prove a repair is correct.

For JSON/SARIF output and more examples, see the [usage guide](https://github.com/taipei49314/checkwash/blob/v0.7.0/docs/releases/v0.7.0-public-launch.md#use-it-and-preserve-the-result).

## Know the limits

**v0.7.0 is alpha.** A pass does not prove that a change is correct or honest.
Python is the main language supported; JS/TS support covers a limited set of
test patterns. Known gaps remain.

Legitimate refactors can be flagged. The frozen historical corpus records
**22 blocks out of 60 (36.7%)**. A separate pre-release source replay records
4/60 blocks, including 2/57 strictly qualified cases; three existing fixture
errors remain. These are selected examples, not a general false-positive rate.

A changed file over 1,000,000 bytes is not read. An image, data file,
lockfile, document or non-Python production source that large is reported on
stderr and earns no repair evidence. A file a detector must read (test,
conftest, CI, guardrail, snapshot, JS setup, runner script or test-command
file) that large stops the run with exit 2, naming it. So do a `.py` module
that large, whether added, deleted, renamed or edited, and a pytest
configuration file (`pytest.ini`, `.pytest.ini`, `pytest.toml`,
`.pytest.toml`, `tox.ini`, `setup.cfg`, `pyproject.toml`) at any depth:
either can change the module a test's import resolves to. So do any
non-generated file under the root `scripts/` directory, read for the pytest
commands it runs, a file with no extension (a dotfile such as `.envrc`
counts), since a shebang the run cannot see could make it a runner script,
and any other changed file that large the analysis needs as context.
Generated output (`build/`, `dist/`, `node_modules/` and the like) is set
aside like a production source unless it is a Python module, a pytest
configuration, a CI file, a runner script or a file with no extension. A root
`poetry.lock` or `uv.lock` that large at the base is skipped with a note; a
root `pyproject.toml` or `requirements*.txt` that large at the base stops
every run with exit 2, naming it.

Try it on your own changes before making it required. [Coverage and limitations](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md#coverage-and-adoption-cost)
· [Known gaps](https://github.com/taipei49314/checkwash/blob/main/docs/adversarial-catalog-2026-09.md)

## Use it in CI

To stop a merge, make the **`checkwash` status check required** in your
repository's branch rules. Installing the tool or adding a workflow alone
does not enforce its verdict.

The recommended Action and the CLI above are both **v0.7.0**.
Record which version you use. [Full setup and exemptions](https://github.com/taipei49314/checkwash/blob/main/docs/enterprise.md)

**Advisory: Action trust boundary.** Every published Action release up to and
including v0.7.0 is affected, including the recommended v0.7.0 pin above. It runs `python -m pip` and then `checkwash` with the
pull request's checkout as the working directory, so a module in that checkout
(for example a top-level `pip` package) can replace the installer and then the
checker, and the pull request can tamper with the result it is judged by. Under
checkwash's threat model that applies to every pull request it judges.
[#372](https://github.com/taipei49314/checkwash/pull/372) fixes it on main; it is
not released yet. Until it is, treat a pass as advisory, review the pull request
before merging and keep `comment-pr` off. After the next release is published,
the README on main will recommend that release's Action instead of the one
before it. [Known issue](https://github.com/taipei49314/checkwash/blob/main/SECURITY.md#known-issue-action-trust-boundary)

<a id="required-check--the-only-configuration-that-blocks-a-merge"></a>
<details>
<summary>Copy the GitHub Actions workflow and require the check</summary>

Save this as `.github/workflows/checkwash.yml`:

```yaml
# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@d647a65e9734f2b6e57b528ea595b1507dc31094 # v0.7.0
```

After the workflow runs, open **Settings → Rules → Rulesets** and require
the `checkwash` status context. Keep the job unconditional.

If you administer the repository and have this project's ruleset file,
you can instead create the rule with:

```bash
gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json
```

This adds a ruleset; it does not replace existing ones. `checkwash doctor`
can inspect the local workflow, but cannot verify live branch protection.

**Why the older Action pin?** A release cannot embed its own commit SHA,
so the documented Action adopts a verified pin from the prior release.
The recommended v0.7.0 Action includes the engine published on 2026-10-12.
A CLI upgrade does not update an existing Action. To verify another trusted release, use
`git rev-parse 'vX.Y.Z^{commit}'`.
[Action reference](https://github.com/taipei49314/checkwash/blob/main/action/README.md)

</details>

## More options and evidence

<a id="install"></a>
<details>
<summary>Install as a CLI or try the included examples</summary>

If you already use pipx, install the fixed version:

```bash
pipx install checkwash==0.7.0
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.7.0

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline
```

`checkwash demo` replays eight real tampering cases and one honest fix.
It illustrates known patterns; it is not a coverage guarantee.
You can run the same examples with `python checkwash.pyz demo`.

</details>

<a id="measured-not-asserted"></a>
<details>
<summary>Read the measurements and their limits</summary>

The historical six-repo sweep recorded **46 / 1800 = 2.56%** blocks:
**31 false positives (1.72%)**, 15 legitimate
  policy blocks (0.83%). The tracked artifacts record engine **v0.3.0**
(the release commit, swept 2026-09-07) on a corpus used to tune the
detectors, not a held-out result.

**1.33% of the corpus (24/1800) records opaque production changes**.
That flag does not establish that each verdict changed or each diff was unanalyzed.

Review methods vary: the three-rater agreement study covers an older
35-diff cohort, not all 46 blocks. The dedicated **22/60 refactor** result
(`benchmarks/refactors/results-latest.json`, a dated v0.3.1 snapshot) is a
separate population; the general-commit rate does not predict it.

[Measurements and source data](https://github.com/taipei49314/checkwash/blob/main/benchmarks/README.md)
· [Generated results](https://github.com/taipei49314/checkwash/blob/main/benchmarks/RESULTS.md)
· [Failure ledger](https://github.com/taipei49314/checkwash/blob/main/benchmarks/FAILURES.md)

</details>

| Looking for… | Start here |
|---|---|
| Installation checks, versions and first use | [v0.7.0 release guide](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.7.0-public-launch.md) |
| JSON/SARIF contracts and upgrades | [Stability](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md) |
| Assertion support, coverage warnings and artifact checks | [Assertion coverage](https://github.com/taipei49314/checkwash/blob/main/docs/assertion-coverage.md) |
| Required checks and reviewed exemptions | [Enterprise setup](https://github.com/taipei49314/checkwash/blob/main/docs/enterprise.md) |
| Contributing or reporting a problem | [Contributing](https://github.com/taipei49314/checkwash/blob/main/CONTRIBUTING.md) · [Issues](https://github.com/taipei49314/checkwash/issues) · [Security reports](https://github.com/taipei49314/checkwash/blob/main/SECURITY.md) |
| Readiness for 1.0 | [Criteria — not met](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md#what-must-change-before-10) |

Related projects: [checkwash-corpus](https://github.com/taipei49314/checkwash-corpus)
and [smallestlie](https://github.com/taipei49314/smallestlie) hold evaluation work.

The package also includes a limited quality preview for coverage, Ruff and
mypy configuration review. [Setup and CI adoption guide](https://github.com/taipei49314/checkwash/blob/main/docs/quality-adoption.md).
Its frozen legacy-byte comparison remains red; publication does not establish
natural-case acceptance or effective enforcement.

Alpha pre-release. 22 detectors, 18394 tests in the current source tree.
Zero runtime dependencies. [Apache-2.0](https://github.com/taipei49314/checkwash/blob/main/LICENSE).
