Metadata-Version: 2.4
Name: honestly
Version: 0.4.1
Summary: Audit how often your coding agent claims work it never verified. Reads Claude Code and Codex session logs. Deterministic rules, no LLM.
License-Expression: MIT
Project-URL: Homepage, https://github.com/kennywu411-dev/honestly
Project-URL: Repository, https://github.com/kennywu411-dev/honestly
Project-URL: Issues, https://github.com/kennywu411-dev/honestly/issues
Keywords: claude-code,codex,ai-agents,hooks,verification,developer-tools
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# honestly

[![tests](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml/badge.svg)](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/honestly)](https://pypi.org/project/honestly/) ![python](https://img.shields.io/badge/python-3.9%2B-blue) ![deps](https://img.shields.io/badge/dependencies-none-success)

**Coding agents report "tests pass" without always running them. `honestly` checks whether a matching test or build actually ran after the last code edit, and blocks the claim when it did not.**

![honestly guard stopping a stale "tests pass" claim](assets/demo.gif)

Claude Code and Codex edit code, then report *tests pass*, *fixed*, *build ok*, *pushed*. Some of those reports were never run: the agent remembers an earlier pass, edits again, and repeats the old result as current. The reply looks the same either way, so the error surfaces later, in CI, on a colleague's machine or in production, where it costs hours instead of seconds. `honestly` moves the check back to the cheapest moment: before the sentence is sent.

## Install the guard

```bash
pip install honestly          # Python 3.9+, standard library only, nothing leaves your machine
honestly guard install        # adds two hooks to ~/.claude/settings.json (--dry-run shows them first)
```

From then on, when the agent is about to end a turn with an unverified claim, this appears in the chat, in the language of the conversation:

```
honestly stepped in: Claude Code (claude-opus-5-5) in project docpilot, answering "fix the login bug"
said "Fixed the login bug, all 12 tests pass" but ran no check after editing login.py.
It has been told to run one first.
```

The agent then runs the check and starts its next reply by owning up: *"honestly caught me: I said tests pass before running them; I have now run them and the result is 12 passed."* If the check fails, it sees the failure and fixes the code. If no check can be run, it must rewrite the sentence as unverified. The second hook refuses a test or build command whose result is cut off by `| tail` or `| head`, since a failure would look like a pass.

| Hook | When | What it does |
|---|---|---|
| `Stop` | the agent is about to end its turn | The reply claims *tests pass / fixed / pushed …* but this turn shows no matching check after the last code edit: the turn is blocked once, with the sentence and the missing step. |
| `PreToolUse` (Bash) | before a command runs | A test or build piped through `tail`/`head` without `set -o pipefail` loses its exit code. The call is denied with the one-line fix. |

```bash
honestly catches              # what guard stopped in the last 7 days, by project and model
honestly guard status         # or: uninstall (removes exactly the entries it added; a backup is kept)
honestly guard notify on      # optional: also show each catch as a desktop notification (off to stop)
```

If your project already has a `verify` skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.

## Audit your history

```bash
honestly audit                # reads ~/.claude/projects (Claude Code) and ~/.codex/sessions (Codex)
honestly audit --since 30d    # last 30 days only (also 8w, 3m)
honestly explain 3            # the full timeline behind the 3rd flagged claim: edit, tools after it, why
honestly share                # writes a card to ~/.honestly/ (SVG, plus PNG when Chrome, Edge or rsvg-convert is present)
```

![honestly audit on the author's sessions](assets/hero.png)

```
Verification rate: 95.2%   (278 verified / 292 decided claims)
Sessions: 101   Claims about this turn's work: 362

  verified         278   the check or action behind the claim is on the transcript and succeeded
  unverified        14   it claimed success (tests pass, fixed, sent, deployed, saved...) with nothing behind it
  contradicted       0   the check or action it ran says otherwise
  could not check   63   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
  weak               7   a visual fix judged by looking at a rendered image; not counted either way
```

Those are the author's own numbers from 101 sessions, with every flag confirmed by a second, independent rater. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found [80 with no run behind them](https://github.com/anthropics/claude-code/issues/104504). Benchmarks report the same on ML tasks ([57% of frontier-agent runs take a planted shortcut](https://arxiv.org/abs/2608.30724), and telling the agent not to barely helps). Light users will see few claims at first; the guard works from the first session regardless.

- **Deterministic.** Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
- **Precision first.** Anything it cannot see is `could not check`, never `unverified`. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag ([`gate/REPORT.md`](gate/REPORT.md)).
- **Reads the output, not just the exit code.** `pytest -q | tail -3` returns tail's exit code; `honestly` reads the last `N passed / N failed` summary and reports `contradicted` when it disagrees with the sentence.

## Beyond code: sent, deployed, downloaded, saved, checked, scheduled

Agents also say *I've emailed the maintainer*, *deployed to production*, *all 10 files downloaded*, *saved to report.csv*, *I checked the docs*, *scheduled the job*. These have no code edit to anchor on, so the evidence window is the whole turn: did a tool call or command that performs the action run before the sentence, and did it succeed?

| Claim | Counts as evidence |
|---|---|
| sent / emailed / replied / "sent you the image" | a send, reply or SendUserFile tool call; `gh pr comment`, sendmail |
| deployed / published / is live | vercel, fly, kubectl apply, docker push, firebase deploy, `gh release create`, twine upload… |
| downloaded | curl -o, wget, yt-dlp, git clone, hf_hub_download, brew install… or a listing that shows the files |
| saved / exported to *file* | a Write/Edit of that file, a redirect or `-o` naming it |
| checked / looked it up / 核对过 | Read, Grep, WebSearch, WebFetch, a browser tool, grep/curl/gh/osascript… |
| scheduled | CronCreate, a scheduled-task tool, crontab, launchctl |
| verified / 跑通了 / 实测通过 | any run or check above; a sentence that names a tool (docker, pytest…) needs that tool |

Precision rules learned from labelling 41 flags with a second independent rater (35 of them were summaries, background jobs or earlier turns): a turn with no tool calls is a summary and is never flagged (`no edit this turn`); the previous turn's tools count, since replies often open by reporting what the last turn did; a background task, subagent, browser action or another session makes the claim `could not check`; "checked" is never `contradicted` by one failed lookup. On the author's logs the seven new categories now yield 134 verified, 46 could not check and 0 flags, so what they catch on other people's logs is still an open question.

## How a claim is judged

1. **Scope: one turn.** Everything the agent did between your prompt and its reply.
2. **Last code edit.** `Edit`/`Write` tools, or `sed -i`, `cat > file`, heredocs that write files. Edits to docs, logs, notes, memory, `.env`, temp dirs or `.agent/` state do not count.
3. **Matching check after it.** Tests for *tests pass*; lint/type/build/check tools for *check passed*; render tools for *rendered*; `git push` / `kaggle … push` for *pushed*; tests, checks or renders for *fixed* (a generic script run is not evidence that a fix works).
4. **Its output.** The last summary line decides: `12 passed` backs the claim, `2 failed` contradicts it, `fail 0` is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (`pytest -k`, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ `13 passed`).
5. **Visual fixes.** "Fixed the overlap" followed by a render and a look at the screenshot is `weak`: shown separately, counted neither as verified nor unverified.
6. **Everything invisible → `could not check`**: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
7. **Not a claim:** plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.

Forked sessions are de-duplicated so the same turn is counted once.

## Precision

Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.

| Round | Flagged `unverified` | Author precision | Independent precision | What was fixed next |
|---|---|---|---|---|
| 1 | 38 | 56% | 69% | doc/log/memory edits counted as code edits; versioned `hyperframes@x check` missed |
| 2 | 22 | 53% | 52% | variable paths, `.agent/state.json`, heredoc bodies; judge output by the last summary line |
| 6 | 12 | 12/12 | 11/11 (+1 unsure) | quote-aware multi-line splitting; writing heredocs never count as verification |

Rounds 3–5 were intermediate fixes (heredoc parsing, variable paths, remote-run claims) and are listed in the report. The independent rater is a Claude agent in a fresh context with no access to the author's labels; it is still a model, not a human. A human second rater is the obvious next step, and the labelling sheets are available on request.

**Recall.** Two raters (author + independent agent) then labelled 66 claims the audit had *not* flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. `honestly` still prefers `could not check` over a wrong flag, and the full labelling material is described in [`gate/REPORT.md`](gate/REPORT.md).

## What it is not

- Not a judge of *correctness*. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
- Not a leaderboard of models. The per-model table is your sessions only.
- Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
- The guard is Claude Code only: Codex has no equivalent stop-before-reply hook, so Codex sessions can be audited but not intercepted.
- Not a judge of test *adequacy*. It checks that a matching check ran after the edit and what its output said, not whether that check covers the lines that changed.
- Subagent and background runs are reported as `could not check`; they are not read yet.
- Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle) and then against two external testers' logs, which is where the first false flags came from (an English bibliography title containing "fixed bed reactor"). Other stacks are covered by synthetic tests only; expect to report a false flag or two, and they are fixed the same day. Previously published as `sober-cli`.

## Related work

[claude-guard-hooks](https://github.com/jefftanlearn-eng/claude-guard-hooks) and [groundtruth](https://github.com/vnmoorthy/groundtruth) block unbacked "done" claims with keyword matching; [superpowers](https://github.com/obra/superpowers) asks for verification in a skill (prompt-level). `honestly` differs in reading command *output*, scoring your history across agents and models, and refusing to count what it cannot see.

中文说明见 [README.zh-CN.md](README.zh-CN.md)。
