Metadata-Version: 2.5
Name: checkwash
Version: 0.9.0
Summary: Flags known patterns of weakened tests and CI checks in a Git diff.
Project-URL: Homepage, https://github.com/taipei49314/checkwash
Author: taipei49314
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agent,ci,code-review,reward-hacking,test-tampering
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# checkwash

**v0.8.0's Action no longer runs code from the pull request it judges; commands
run from a subdirectory read the repository from its root; and oversized files
that no detector reads are carried unread instead of stopping the run.**
[Release evidence and known limits](https://github.com/taipei49314/checkwash/blob/v0.8.0/docs/releases/v0.8.0-public-launch.md).
A pass does not prove correctness; retain exit 2 and diagnostics as analysis
errors. With `--format hook-json` (the Claude Code Stop hook) an engine error
now exits 0 with a "not analysed" message. The recommended Action is pinned to
the same v0.8.0 release.

**Development snapshots:** the source in this checkout may be newer than the
published CLI. [The advertised-release manifest](https://github.com/taipei49314/checkwash/blob/main/release/advertised.toml)
records the version, commit and download bytes behind the installation commands
below. New changes require their own candidate evidence and remain unreleased
until the publication record verifies them; candidate CI success does not mean
that the recommended Action pin includes them.

[![CI](https://github.com/taipei49314/checkwash/actions/workflows/ci.yml/badge.svg)](https://github.com/taipei49314/checkwash/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/checkwash.svg)](https://pypi.org/project/checkwash/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/taipei49314/checkwash/blob/main/LICENSE)

**Check whether a code change weakens your tests.**

A test can pass after it stops checking something useful. checkwash reads
your Git diff and flags known patterns such as deleted assertions, skipped
tests, relaxed expectations, and CI checks that stop failing.

```diff
- assert total == 105.3
+ assert total > 0
```

The new assertion accepts many incorrect totals. checkwash can flag changes
like this for review, including changes written by coding agents.

**Runs locally. No LLM. No network during analysis. Never executes your code.**

**v0.3.0 replaces five file-wide exemption rules with content-bound ones.**
In v0.2.13 those rules could reuse a recorded approval for a later change to
the same path; v0.3.0 retires their path-only keys. The [upgrade notes](https://github.com/taipei49314/checkwash/blob/main/docs/remediation-upgrade.md)
describe the content-bound replacement, retired keys, and installation fixes.

**v0.4.0 extends detection of runtime suppression, collection changes,
expectation rewrites and substituted subjects, and recognizes more table refactors.** The proofs retain
input identity, execution context and conservative handling of unknown code.
[Release evidence and remaining limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.0-public-launch.md)

**v0.4.1 repairs Node assertion recognition ([#164](https://github.com/taipei49314/checkwash/issues/164))**
and covers Node test-file names, bounded import aliases and lexical shadows.
Coverage warnings expose known assertion candidates the scanner cannot represent;
they do not prove complete JS/TS coverage. The same assertion contract checks
the source, wheel and zipapp. [v0.4.1 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.1-public-launch.md)

**v0.4.2 strengthens JS/TS assertion evidence.** Test callbacks own their
assertions; changing a scalar expected value or loosening positive
`toBeCloseTo` precision reaches the existing detectors. Equivalent scalar
spellings remain equivalent across subject checks. This is bounded syntax
support, not general JS/TS analysis. [v0.4.2 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.4.2-public-launch.md)

**v0.5.0 closes a round of red-team reports (#172–#181).** A skip in a
fixture or setup the test runs, a selector over edited tests, a package
manifest's test command, Jest's `__tests__/` layout, JS block-level and
imperative skips, JS module mocks, hand-rolled JS tolerances, chai assertions
and CI control flow that stops a runner now reach the detectors; #173, #180
and #181 are closed in part. Some existing findings now carry a different fingerprint,
so allowlist entries for those shapes must be re-recorded; the JSON shape is
unchanged. [v0.5.0 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.5.0-public-launch.md)

**v0.6.0 works through the reports and rulings that followed (#196–#261).**
JS/TS assertions keep their meaning across Jest, chai, Vitest and node:test,
and the runner a file's imports name decides whether a moved test still runs
and what a focus stops. Hand-rolled tolerances in both frontends, pytest
selectors under explicit targets, skip guards in a body, a setup or a conftest
hook, every `pytestmark` spelling and a CI config the reader cannot read now
reach the detectors. Some existing findings now carry a different fingerprint,
so allowlist entries for those shapes must be re-recorded; the JSON shape is
unchanged. [v0.6.0 evidence and limits](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.6.0-public-launch.md)

**The next minor (unreleased) makes assertion-level exemptions content-bound
([#421](https://github.com/taipei49314/checkwash/issues/421)).** Through
v0.8.0, an exemption for one rewrite of an assertion also allowed every later
rewrite of the same line. Nine assertion-level rules now key a finding by the
old and the new assertion as well, TEST_DISABLED keys one by what the change
disabled ([#512](https://github.com/taipei49314/checkwash/issues/512)), and
their old keys stop matching: re-review and re-record those entries. [Assertion-level notes](https://github.com/taipei49314/checkwash/blob/main/docs/remediation-upgrade.md#assertion-level-rules-next-minor-421)

## Try it

You need **Python 3.11+ and Git**. Download and try the offline examples first.

Windows PowerShell (including 5.1):

```powershell
curl.exe -LO https://github.com/taipei49314/checkwash/releases/download/v0.8.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo
```

PowerShell 5.1 aliases `curl` to `Invoke-WebRequest`; use `curl.exe` as written.
macOS/Linux or Git Bash:

```bash
curl -LO https://github.com/taipei49314/checkwash/releases/download/v0.8.0/checkwash.pyz
python checkwash.pyz --version
python checkwash.pyz demo
```

Then, from the root of a repository with at least two commits:

```bash
python checkwash.pyz check HEAD~1..HEAD
```

This checks your last commit. Start with a change you already understand.
You can also [download the file in your browser](https://github.com/taipei49314/checkwash/releases/download/v0.8.0/checkwash.pyz).
For uncommitted changes use `python checkwash.pyz check`. For a branch review,
use `python checkwash.pyz check BASE...HEAD` to compare from the merge base;
`BASE..HEAD` compares the two named snapshots directly.

From a subdirectory, `check` analyses the whole repository from its Git root
and says so in one line on stderr (since v0.8.0,
[#416](https://github.com/taipei49314/checkwash/issues/416)). Before v0.8.0,
run it from the repository root.

| Result | What to do |
|---|---|
| **Pass · exit 0** | No finding requires blocking under your configuration. Keep running your normal tests and review. |
| **Block · exit 1** | Read the finding and the diff. It may be weakened verification or a false positive. |
| **Error · exit 2** | Resolve the input or analysis error before relying on the result. |

The default threshold is **high**: a visible **warn** can still pass.
`REPAIR_EVIDENCE` describes related changes in the same diff; it does not
prove a repair is correct.

For JSON/SARIF output and more examples, see the [usage guide](https://github.com/taipei49314/checkwash/blob/v0.8.0/docs/releases/v0.8.0-public-launch.md#use-it-and-preserve-the-result).

## Know the limits

**v0.8.0 is alpha.** A pass does not prove that a change is correct or honest.
Python is the main language supported; JS/TS support covers a limited set of
test patterns. Known gaps remain.

In v0.8.0, JS/TS `test.each`, `it.each` and `test.for` tables are not
scanned: an assertion weakened or deleted in one, an edited row or a deleted
table passes. From the next minor (unreleased), each such table is a test
unit, and its rows are read when they can be counted (an inline array or a
tagged template, or a same-file `const` read only as a table). The rows of a
`describe.each` table stay unread, and a table with a computed title is not a
unit ([#428](https://github.com/taipei49314/checkwash/issues/428)).

Legitimate refactors can be flagged. The frozen historical corpus records
**22 blocks out of 60 (36.7%)**. A separate pre-release source replay records
4/60 blocks, including 2/57 strictly qualified cases; three existing fixture
errors remain. These are selected examples, not a general false-positive rate.

A changed file over 1,000,000 bytes is not read. An image, data file,
lockfile, document or non-Python production source that large is reported on
stderr and earns no repair evidence. From the next minor (unreleased,
[#454](https://github.com/taipei49314/checkwash/issues/454)), `--format json`
also lists it under `unread`, SARIF names it in a tool execution
notification, and the term report adds a `!` line and, when there is no
finding, opens with "analysis incomplete"; the Stop hook's JSON and
`--emit-ir` do not name it. A file a detector must read (test, conftest, CI,
guardrail, snapshot, JS setup, runner script or test-command file) that large
stops the run with exit 2, naming it. So do a `.py` module
that large, whether added, deleted, renamed or edited, and a pytest
configuration file (`pytest.ini`, `.pytest.ini`, `pytest.toml`,
`.pytest.toml`, `tox.ini`, `setup.cfg`, `pyproject.toml`) at any depth:
either can change the module a test's import resolves to. So do any
non-generated file under the root `scripts/` directory, read for the pytest
commands it runs, and any other changed file that large the analysis needs
as context. A shell shebang gives a file the shape of a runner script, whatever
its name, except a `.py` module or a JS/TS test file.
From the next minor (unreleased,
[#455](https://github.com/taipei49314/checkwash/issues/455)), checkwash reads
the first 200 bytes of a side over the limit to tell, and uses them for
nothing else; a Git side is streamed and checked against its object id.
Where those bytes are read, a file that large whose side starts with a shell
shebang stops the run, naming that side, whatever its suffix (a `.py` module
or a JS/TS test file that large stops the run anyway, by its own rule). A
file with no extension (a dotfile such as `.envrc` counts) without one, such
as a `README` or a `LICENSE`, is set aside like any document, unless another
rule above makes it one a detector must read. v0.8.0 reads no byte of such a
side, so every file with no extension that large stops the run there. On main
a Git side over 64,000,000 bytes is not streamed, so its first bytes are not
known: a file with no extension still stops the run when one of its Git sides
is that large, and a suffixed file whose shebang is only on such a side is
still set aside.
When a file that large stops the run, from the next minor (unreleased)
`--format json` prints `{"unread": [...]}` on stdout instead of nothing: one
`unread` entry, for the side the run refused. It is not a findings document,
and term and SARIF output stay empty. With a range, json output stays empty
too when the refused file is a symlinked pytest configuration's target that
no detector must read.
Generated output (`build/`, `dist/`, `node_modules/` and the like) is set
aside like a production source unless it is a Python module, a pytest
configuration, a CI file, a runner script or a file with no extension whose
shebang is not known. A root
`poetry.lock` or `uv.lock` that large at the base is skipped with a note on
stderr, and the skip is not listed under `unread`; when the diff also changes
that lockfile, the change's sides over the limit are listed like any other. A
root `pyproject.toml` or `requirements*.txt` that large at the base stops
every run, naming it: term, json and SARIF output exit 2 and name it on
stderr only. The Claude Code Stop hook (`--format hook-json`) reports an
engine error, this one included, as a non-blocking "not analysed" message
on stdout that quotes the error, and exits 0.
From the next minor (unreleased), when the run stops on a file the hook's diff
changes and that file's working-tree side is over the limit, the hook blocks
the stop instead, naming the file and asking to shrink, split or revert it
([#456](https://github.com/taipei49314/checkwash/issues/456)). A file whose
base side alone is over the limit, a root manifest over it at the base, and a
padded file committed before the stop keep the message; the full list is in
[subject integrity](https://github.com/taipei49314/checkwash/blob/main/docs/subject-integrity.md).

Try it on your own changes before making it required. [Coverage and limitations](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md#coverage-and-adoption-cost)
· [Known gaps](https://github.com/taipei49314/checkwash/blob/main/docs/adversarial-catalog-2026-09.md)

## Use it in CI

To stop a merge, make the **`checkwash` status check required** in your
repository's branch rules. Installing the tool or adding a workflow alone
does not enforce its verdict.

The recommended Action and the CLI above are both **v0.8.0**.
Record which version you use. [Full setup and exemptions](https://github.com/taipei49314/checkwash/blob/main/docs/enterprise.md)

**Advisory: Action trust boundary.** Every published Action release up to and
including v0.7.0 runs `python -m pip` and then `checkwash` with the pull
request's checkout as the working directory, so a module in that checkout (for
example a top-level `pip` package) can replace the installer and then the
checker, and the pull request can tamper with the result it is judged by. Under
checkwash's threat model that applies to every pull request it judges. v0.8.0
fixes it ([#372](https://github.com/taipei49314/checkwash/pull/372)); pin the
v0.8.0 Action below. If you stay on v0.7.0 or earlier, treat a pass as
advisory, review the pull request before merging and keep `comment-pr` off.
[Known issue](https://github.com/taipei49314/checkwash/blob/main/SECURITY.md#known-issue-action-trust-boundary)

<a id="required-check--the-only-configuration-that-blocks-a-merge"></a>
<details>
<summary>Copy the GitHub Actions workflow and require the check</summary>

Save this as `.github/workflows/checkwash.yml`:

```yaml
# .github/workflows/checkwash.yml
on: [pull_request]

permissions:
  contents: read

jobs:
  checkwash:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: taipei49314/checkwash/action@f02b5d8a07fa39ee1898881f66c571eb164cb11c # v0.8.0
```

After the workflow runs, open **Settings → Rules → Rulesets** and require
the `checkwash` status context. Keep the job unconditional.

If you administer the repository and have this project's ruleset file,
you can instead create the rule with:

```bash
gh api repos/OWNER/REPO/rulesets --method POST --input action/required-ruleset.json
```

This adds a ruleset; it does not replace existing ones. `checkwash doctor`
can inspect the local workflow, but cannot verify live branch protection.

**Why is the pin added after the release?** A release
cannot embed its own commit SHA, so a release commit carries the prior
release's pin; once a release's publication is verified, a follow-up on main
recommends that release's own peeled commit. The recommended v0.8.0 Action includes the engine
published on 2026-10-15. A CLI upgrade does not update an existing Action. To verify another trusted release, use
`git rev-parse 'vX.Y.Z^{commit}'`.
[Action reference](https://github.com/taipei49314/checkwash/blob/main/action/README.md)

</details>

## More options and evidence

<a id="install"></a>
<details>
<summary>Install as a CLI or try the included examples</summary>

If you already use pipx, install the fixed version:

```bash
pipx install checkwash==0.8.0
# or from the release tag:
pipx install git+https://github.com/taipei49314/checkwash@v0.8.0

checkwash check HEAD~1..HEAD
checkwash demo                  # 8 real tampering cases, blocked, offline
```

`checkwash demo` replays eight real tampering cases and one honest fix.
It illustrates known patterns; it is not a coverage guarantee.
You can run the same examples with `python checkwash.pyz demo`.

</details>

<a id="measured-not-asserted"></a>
<details>
<summary>Read the measurements and their limits</summary>

The historical six-repo sweep recorded **46 / 1800 = 2.56%** blocks:
**31 false positives (1.72%)**, 15 legitimate
  policy blocks (0.83%). The tracked artifacts record engine **v0.3.0**
(the release commit, swept 2026-09-07) on a corpus used to tune the
detectors, not a held-out result.

**1.33% of the corpus (24/1800) records opaque production changes**.
That flag does not establish that each verdict changed or each diff was unanalyzed.

Review methods vary: the three-rater agreement study covers an older
35-diff cohort, not all 46 blocks. The dedicated **22/60 refactor** result
(`benchmarks/refactors/results-latest.json`, a dated v0.3.1 snapshot) is a
separate population; the general-commit rate does not predict it.

[Measurements and source data](https://github.com/taipei49314/checkwash/blob/main/benchmarks/README.md)
· [Generated results](https://github.com/taipei49314/checkwash/blob/main/benchmarks/RESULTS.md)
· [Failure ledger](https://github.com/taipei49314/checkwash/blob/main/benchmarks/FAILURES.md)

</details>

| Looking for… | Start here |
|---|---|
| Installation checks, versions and first use | [v0.8.0 release guide](https://github.com/taipei49314/checkwash/blob/main/docs/releases/v0.8.0-public-launch.md) |
| JSON/SARIF contracts and upgrades | [Stability](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md) · [Embedding guide](https://github.com/taipei49314/checkwash/blob/main/docs/embedding.md) · [Findings schema](https://github.com/taipei49314/checkwash/blob/main/docs/findings.schema.json) |
| Assertion support, coverage warnings and artifact checks | [Assertion coverage](https://github.com/taipei49314/checkwash/blob/main/docs/assertion-coverage.md) |
| Required checks and reviewed exemptions | [Enterprise setup](https://github.com/taipei49314/checkwash/blob/main/docs/enterprise.md) |
| Contributing or reporting a problem | [Contributing](https://github.com/taipei49314/checkwash/blob/main/CONTRIBUTING.md) · [Issues](https://github.com/taipei49314/checkwash/issues) · [Security reports](https://github.com/taipei49314/checkwash/blob/main/SECURITY.md) |
| Readiness for 1.0 | [Criteria — not met](https://github.com/taipei49314/checkwash/blob/main/docs/stability.md#what-must-change-before-10) |

Related projects: [checkwash-corpus](https://github.com/taipei49314/checkwash-corpus)
and [smallestlie](https://github.com/taipei49314/smallestlie) hold evaluation work.

The package also includes a limited quality preview for coverage, Ruff and
mypy configuration review. [Setup and CI adoption guide](https://github.com/taipei49314/checkwash/blob/main/docs/quality-adoption.md).
Its frozen legacy-byte comparison remains red; publication does not establish
natural-case acceptance or effective enforcement.

Alpha pre-release. 22 detectors, 20488 tests in the current source tree.
Zero runtime dependencies. [Apache-2.0](https://github.com/taipei49314/checkwash/blob/main/LICENSE).
