Metadata-Version: 2.5
Name: hearsay-pr
Version: 0.6.0
Summary: Deterministic fact-checking for AI-agent PR descriptions. No LLM, offline, zero runtime deps. (CLI command: hearsay)
Project-URL: Homepage, https://github.com/hearsay-dev/hearsay
Project-URL: Repository, https://github.com/hearsay-dev/hearsay
Project-URL: Changelog, https://github.com/hearsay-dev/hearsay/blob/main/CHANGELOG.md
Author: hearsay contributors
License: MIT
License-File: LICENSE
Keywords: agents,ai,ci,code-review,fact-check,llm,mutation-testing,pull-request,slop,static-analysis,test-quality
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Description-Content-Type: text/markdown

# 🔥 hearsay

[![CI](https://github.com/hearsay-dev/hearsay/actions/workflows/ci.yml/badge.svg)](https://github.com/hearsay-dev/hearsay/actions/workflows/ci.yml)
[![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![calibration](https://img.shields.io/badge/fire%20precision-calibrating-orange)](docs/CALIBRATION.md)

**PR descriptions are hearsay. Verify them.**

Coding agents write their own PR descriptions — and then reviewers merge based
on them. *"Updated all call sites." "Added tests for X." "No breaking
changes."* These are self-reported claims from the same model that wrote the
code, and they are systematically overconfident. hearsay extracts every
checkable claim from a PR description and verifies it against the actual diff
with **deterministic static analysis**: regex, git plumbing, and the stdlib
`ast` module.

**No LLM. No API keys. No network. Zero runtime dependencies.**

```text
$ hearsay demo
hearsay — auditing 6 claim(s) in the PR description
base main (8b84bf5) → head HEAD (a261157) · 2 file(s) changed (+14/−1)

🔥 PANTS ON FIRE  “Renamed `greet` to `hello` and updated all call sites.”
   claimed complete, but 2 stale reference(s) to `greet` remain (`greet` → `hello`)
     - app.py:4  `print(greet("world"))`
     - test_greet.py:6  `greet.hello("world")`

❌ CONTRADICTED  “- Added tests for `hello`”
   1 test(s) exist but none of them can fail: test_greet.py: `test_greets_politely` — contains no assertions — it cannot fail

🔥 PANTS ON FIRE  “- No breaking changes”
   renamed public symbol(s) `greet` → `hello` without updating 2 call site(s)
     - app.py:4  `print(greet("world"))`

🔥 PANTS ON FIRE  “- Docs updated”
   the diff touches no documentation files (*.md, *.rst, docs/, CHANGELOG) — nothing was documented

⚠️ UNVERIFIABLE  “- Improved performance for hot paths”
   performance claims need benchmarks — no number, no claim

⚠️ UNVERIFIABLE  “Fixes #42”
   issue #42 cannot be fetched offline; the diff includes test changes, which is consistent with a fix

──────────────────────────────────────────────────────────────────────────────
6 claims audited: 1 contradicted · 3 pants-on-fire · 2 unverifiable
```

That whole audit ran offline against a synthetic repo in ~40 ms.

## Why

- Agent-generated PR volume is exploding, and reviewers increasingly rubber-stamp
  the description instead of reading the diff.
- Research confirms the mismatch is systematic (see e.g. arXiv:2601.04886 on
  message–code inconsistency in agent commits).
- LLM-based reviewers can't be trusted to catch their own family's spin, cost
  money per call, and aren't reproducible. hearsay is the opposite: the same
  description + the same diff always produce the same verdicts.

## Install

```bash
pipx install hearsay-pr     # or: uv tool install hearsay-pr
# the PyPI distribution is `hearsay-pr`; the CLI command it installs is `hearsay`
# or, from a clone of this repo:
pip install .
# or, for development:
PYTHONPATH=src python -m hearsay demo
```

Requires Python 3.10+ and git. No other dependencies.

## Why deterministic, not LLM?

| | hearsay | LLM PR reviewers |
|---|---|---|
| Verdicts reproducible | ✅ same input → same verdict | ❌ sampled, drifting |
| Cost per PR | **$0.000** (offline, zero deps) | $0.05–0.50 + API keys |
| Can audit its own PRs | ✅ | ⚠️ conflicts of interest |
| Explains every verdict with `file:line` | ✅ | "seems fine overall" |
| Catches agent overclaiming | ✅ its whole purpose | it's the same species |

LLM reviewers are useful; they are just not *evidence*. hearsay is the
deterministic layer underneath them.

## How it works

```
PR description (text)
  │  31 builtin regex patterns ─ active/passive/headline voice
  ▼
structured claims ──► one deterministic checker per claim type
  │                     │ git plumbing (cat-file --batch)
  │                     │ stdlib ast (Python) / symbol tables (JS/TS, Go)
  │                     │ repo-wide stale-reference & API-surface scans
  ▼
verdicts: ✅ corroborated · ❌ contradicted · 🔥 pants-on-fire · ⚠️ unverifiable
  ▼
terminal report · PR comment (markdown) · JSON for your bot
```

## Usage

```bash
# audit a local branch against main, description from a file
hearsay check --base main --desc pr.md

# ...or read the description from stdin (e.g. your editor's PR draft)
hearsay check --base main --desc - < description.md

# ...or pull it straight from GitHub via the gh CLI
hearsay check --pr 123 --base origin/main

# CI gate: exit 2 on pants-on-fire, exit 1 on contradicted
hearsay gate --pr 123 --base origin/main --fail-on contradicted

# post the report as a PR comment (uses gh for auth; hearsay never sees tokens)
hearsay gate --pr 123 --base origin/main --comment

# machine-readable output for your bot
hearsay check --pr 123 --format json
hearsay check --pr 123 --format markdown   # paste into a PR comment

# project setup & introspection
hearsay init        # write a commented hearsay.toml
hearsay patterns    # list the builtin claim patterns
```

## Analysis engines (beyond the PR gate)

The claim gate checks one PR. These three engines audit the whole
repository — they're what makes hearsay a quality engine rather than a
linter for prose:

```bash
hearsay testlint              # six test-smell families, per-test findings
hearsay testlint --fail-on vacuous,no-production   # gate on any mix
hearsay score                 # deterministic test-health score, 0-100, A-D
hearsay score --min 70        # ratchet gate: fail below the bar
hearsay clones                # copy-paste clone clusters (the GitClear #1 metric)
```

`testlint` finds: tests that **cannot fail**, assertions that pin almost
nothing (`is not None`, truthiness-only), byte-identical duplicate test
bodies (agent filler), tests that never touch production code,
sleep-based flaky tests, and public functions **no test references
anywhere**. `score` compiles those into one number with a full breakdown —
every deduction is a counted issue family, never vibes. All engines are
offline, deterministic, and JSON-serializable.

### `hearsay mutiny` — does your test suite actually notice?

Static analysis approximates; mutation testing *proves*. `hearsay mutiny`
applies one deterministic AST mutation at a time to production code
(`+`→`-`, `==`→`!=`, `True`→`False`, dropped return values…), runs your
tests, and lists every change your suite **cannot detect**:

```bash
hearsay mutiny --max-mutants 30 --fail-on-survived
```

Python-only, opt-in (it really runs your tests), and the strongest answer
to "is this test suite real?".

### Type-3 clones

`hearsay clones --normalize-idents` additionally matches clones whose
identifiers were renamed — the agent copy-rename-duplicate signature.

### Extend it: plugins

```toml
# hearsay.toml
plugins = ["hearsay_plugin.py"]
```

A plugin module provides extra `PATTERNS` (claim regexes) and `CHECKERS`
(verdict functions) for existing claim kinds — same contract, same design
rules. See CONTRIBUTING.md.

### Forge-agnostic PR retrieval

`--pr-cmd "glab mr view {pr}"` (or any command with a `{pr}` placeholder
whose stdout is the description) replaces the default `gh` integration —
GitLab, Gitea, or your internal tooling.

### The Hearsay Report generator

`hearsay report --cases cases.jsonl --out report.md` turns labeled
calibration cases into a shareable markdown report — headline precision,
per-claim-kind results, confirmed fire samples. That's the quarterly
publication pipeline, as a command.

## Let the agent police itself (MCP)

`hearsay mcp` serves the engines as MCP tools over stdio — no dependencies,
hand-rolled JSON-RPC. Register it in your MCP client (or just add one line
to AGENTS.md):

```json
{"hearsay": {"command": "hearsay", "args": ["mcp"]}}
```

```markdown
<!-- AGENTS.md -->
Before opening a PR, run hearsay_check on your own description and fix
every pants-on-fire claim. Do not claim work you cannot demonstrate.
```

The agent that wrote the slop becomes the first line of defense against it.

## Fact-check commit messages too

Agents overclaim in commit messages exactly like PR descriptions — and
commit messages can be gated locally, no PR required:

```bash
hearsay check --base main --commits        # audit the branch's messages

# .git/hooks/commit-msg — block provable lies before they land
hearsay commitmsg "$1"
```

`commitmsg` compares the message against the **staged** diff and defaults
to `--fail-on pants-on-fire`: hooks only block what can be proven.

## The ratchet: adopting score on imperfect repos

Absolute thresholds are why quality gates get uninstalled. hearsay score
supports a baseline ratchet instead:

```bash
hearsay score --save-baseline    # writes .hearsay-baseline.json — commit it
hearsay score                    # passes at the baseline, fails on regression
```

Failures show per-issue-kind deltas ("vacuous: 2 → 5"), so the gate tells
you exactly what got worse. Fix forward, re-baseline when you've earned it.

## Configuration

`hearsay init` writes a commented `hearsay.toml`; the same keys work under
`[tool.hearsay]` in pyproject.toml (hearsay.toml wins):

```toml
# hearsay.toml
base = "origin/main"       # CLI --base overrides this
fail_on = "contradicted"   # pants-on-fire | contradicted | unverifiable | none
format = "terminal"        # terminal | markdown | json
exclude = ["**/generated/**", "vendor/**"]   # ignored everywhere
```

Precedence: CLI flag > hearsay.toml > pyproject `[tool.hearsay]` > builtin
default.

## What gets checked

| The claim | The deterministic check |
|---|---|
| "updated all call sites" / "renamed X to Y" | finds removed/renamed symbols in the diff, then scans the whole tree for stale references |
| "added tests for X" | test files must be touched, must reference `X`, and must be able to fail (assertion analysis) |
| "no breaking changes" | public symbols (non-underscore top-level defs) removed or renamed with callers left behind |
| "docs updated" | the diff must actually touch `*.md` / `*.rst` / `docs/` / `CHANGELOG` |
| "fixes #123" | cannot be fetched offline — hearsay says so, and flags the no-tests-in-diff smell |
| "all tests pass" | static analysis cannot run tests — marked unverifiable, honestly |
| "removed dead code" / "cleaned up" | diff must remove lines, and removed symbols must have zero remaining references |
| "refactored X" | pure-addition diffs are not refactors; dangling references are flagged |
| "added a new function `foo`" | `foo` must be defined in the added lines |
| "modified `src/auth.py`" | the named file must actually be touched by the diff |
| "improved performance" / "handles edge cases" / "thread-safe" | collected into the unverifiable list — which is the reviewer's reading list |

### Verdicts

| | Verdict | Meaning |
|---|---|---|
| ✅ | `corroborated` | the diff supports the claim |
| ❌ | `contradicted` | the diff contradicts the claim |
| 🔥 | `pants-on-fire` | claimed done, demonstrably not done (the worst kind of hearsay) |
| ⚠️ | `unverifiable` | cannot be decided statically — hearsay will not pretend otherwise |

The three-valued logic is the point: a checker that never says "unverifiable"
will eventually say "corroborated" about a lie.

## Language support

hearsay is precise where it can be and honest where it can't:

| Tier | Languages | What runs |
|---|---|---|
| **Precise** | Python | all checkers, stdlib-`ast` backed, function-granularity vacuous-test detection |
| **Heuristic** | JavaScript / TypeScript, Go | symbol-level call-site / rename / breaking checks from regex symbol tables; file-granularity test checks (verdict wording always says "heuristic") |
| **Agnostic** | everything else | language-agnostic checkers only: diff-shape claims (removed dead code, refactor), docs, issue refs, bravado collection |

Adding a language is a self-contained `Language(...)` entry plus tests —
a deliberate good-first-issue (see [CONTRIBUTING.md](CONTRIBUTING.md)).

## Use it as a GitHub Action

```yaml
name: hearsay
on: [pull_request]
jobs:
  factcheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }   # hearsay needs the merge base
      - uses: hearsay-dev/hearsay@v0
        with:
          base: origin/main
          fail-on: contradicted    # pants-on-fire always fails
          comment: true            # post the report to the PR
```

Or roll your own step — hearsay is a plain CLI:

```yaml
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hearsay-pr
      - env: { GH_TOKEN: "${{ github.token }}" }
        run: hearsay gate --pr "${{ github.event.pull_request.number }}" --base origin/main
```

Exit codes: `0` pass · `1` contradicted/unverifiable (per `--fail-on`) ·
`2` pants-on-fire · `3` usage or git error.

## Use with pre-commit

```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/hearsay-dev/hearsay
    rev: v0.3.0
    hooks:
      - id: hearsay-gate
        args: ["--base", "origin/main", "--desc", "pr-description.md"]
```

```bash
pre-commit run hearsay-gate --hook-stage manual
```

## Design principles

1. **Deterministic.** Same inputs, same verdicts, always. Auditable regex and
   AST checks — you can read every rule in `src/hearsay/patterns.py` and
   `src/hearsay/checkers.py`.
2. **Offline and free.** Zero runtime dependencies, zero API calls, runs
   anywhere git runs.
3. **Three-valued logic.** "We don't know" is a first-class verdict.
4. **Conservative by default.** Claims that can't be proven false are never
   marked false. `--fail-on` lets you choose how paranoid the gate is.
5. **Evidence or it didn't happen.** Every non-corroborated verdict cites
   `file:line` locations you can click.

## Calibration (why there's no precision badge yet)

Unit tests prove the checkers work on fixtures. They don't prove the
false-positive rate — only labeled real-world PRs do. hearsay ships the
whole calibration pipeline:

```bash
hearsay calibrate --cases cases.jsonl                        # batch predict
hearsay calibrate --cases cases.jsonl --emit-worksheet ws.md # label by hand
hearsay calibrate --cases cases.jsonl --json > results.json  # score
```

Protocol and acceptance thresholds: [docs/CALIBRATION.md](docs/CALIBRATION.md).
Until fire precision ≥ 97% on ≥ 100 labeled claims, the README will not
recommend gating on `contradicted` — and the badge stays honest.

## Roadmap

- [x] Multi-language tiering (Python precise; JS/TS, Go heuristic)
- [x] Calibration pipeline (`hearsay calibrate` + protocol)
- [ ] Reach fire precision ≥ 97% on ≥ 100 labeled real PRs (needs you —
      see [docs/CALIBRATION.md](docs/CALIBRATION.md))
- [ ] Signature-change detection (breaking positional-arg changes)
- [ ] tree-sitter backends to promote more languages to the precise tier
- [ ] `hearsay report` — aggregate scan of N repos → the periodic **Hearsay
      Report** on how often agent PR descriptions survive fact-checking
- [ ] PR-comment bot mode with persistent verdict tables
- [ ] Plugin API for community checkers

## FAQ

**Doesn't the regex extraction miss things?**
Yes — that's why unverifiable exists. hearsay only rules *against* a claim
when a deterministic check proves it, so misses are safe. Template-friendly
descriptions (headings, bullets) extract best.

**Is this accusing agents of lying?**
No — agents are overconfident, not malicious. hearsay treats the description
as untrusted input, exactly like you treat user input.

**Why not just use an LLM reviewer?**
See [Why deterministic, not LLM?](#why-deterministic-not-llm) — reproducibility, cost, and independence. hearsay has no incentive to believe its own kind.

## Development

```bash
pip install -e .[dev]     # or: pip install ruff mypy coverage
ruff check src tests
mypy
python -m unittest discover -s tests -t .   # PYTHONPATH=src if not installed
python scripts/bench.py   # performance smoke test on a synthetic 1200-file repo
```

The full test suite is green, ruff + mypy clean. New contributors:
`CONTRIBUTING.md` doubles as the architecture tour, and `good first issue` =
add a language adapter.

## License

MIT — see [LICENSE](LICENSE).
