Metadata-Version: 2.4
Name: sober-cli
Version: 0.2.1
Summary: Audit how often your coding agent claims work it never verified. Reads Claude Code and Codex session logs. Deterministic rules, no LLM.
License-Expression: MIT
Project-URL: Homepage, https://github.com/kennywu411-dev/sober
Project-URL: Repository, https://github.com/kennywu411-dev/sober
Project-URL: Issues, https://github.com/kennywu411-dev/sober/issues
Keywords: claude-code,codex,ai-agents,hooks,verification,developer-tools
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# sober

[![tests](https://github.com/kennywu411-dev/sober/actions/workflows/tests.yml/badge.svg)](https://github.com/kennywu411-dev/sober/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/sober-cli)](https://pypi.org/project/sober-cli/) ![python](https://img.shields.io/badge/python-3.9%2B-blue) ![deps](https://img.shields.io/badge/dependencies-none-success)

**How often did your coding agent actually verify what it claimed?**

![sober audit on the author's sessions](assets/hero.png)

![sober guard stopping a stale "tests pass" claim](assets/demo.gif)

`sober` reads the session logs that Claude Code and Codex already keep on your machine, finds every sentence where the agent said *tests pass / check passed / fixed / built / rendered / pushed*, and checks whether a matching command really ran **after its last code edit** — and what that command's output said.

```
$ pip install sober-cli && sober audit

Verification rate: 91.5%   (140 verified / 153 decided claims)
Sessions: 96   Claims about this turn's work: 176

  verified         140   a matching check ran after the last code edit and its output agrees
  unverified        13   the agent edited code, then claimed success without running a check
  contradicted       0   the check it ran says otherwise
  could not check   16   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
  weak               7   a visual fix judged by looking at a rendered image; not counted either way

By agent / model:
  claude-code claude-opus-5-5     86%  (131 decided, 29 could not check)
```

Those are the author's own numbers, from 96 real sessions. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found [80 with no run backing them](https://github.com/anthropics/claude-code/issues/104504); most were *stale*: tests passed, the agent edited again, then reported the old pass as current. Benchmarks say the same thing about ML tasks ([57% of frontier-agent runs take a planted shortcut](https://arxiv.org/abs/2608.30724), and telling the agent not to barely helps). `sober` gives you your own number, then stops the pattern with two hooks.

- **Deterministic.** Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
- **Precision first.** Anything it cannot see is `could not check`, never `unverified`. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag (details in [`gate/REPORT.md`](gate/REPORT.md)).
- **Reads the output, not just the exit code.** `pytest -q | tail -3` returns tail's exit code; `sober` reads the last `N passed / N failed` summary instead, and reports a `contradicted` claim when the summary disagrees with the sentence.

## Install

```bash
pip install sober-cli      # Python ≥ 3.9, standard library only
sober audit                # Claude Code (~/.claude/projects) + Codex (~/.codex/sessions)
sober audit --since 30d    # last 30 days only (also 8w, 3m)
sober audit --agent claude-code --json findings.jsonl   # every claim with context; per-project breakdown is in the report
sober explain 3            # the full timeline behind the 3rd flagged claim: edit, tools after it, why
sober share                # writes ~/.sober/sober-card.svg (+ .png via rsvg-convert or a local Chrome)
```

## Make it stop happening: `sober guard`

```bash
sober guard install --dry-run   # shows the hooks it would add to ~/.claude/settings.json
sober guard install
sober guard status | uninstall
```

Two Claude Code hooks, both tiny and deterministic:

| Hook | When | What it does |
|---|---|---|
| `Stop` | the agent is about to end its turn | If the reply claims *tests pass / fixed / pushed …* and this turn shows no matching check after the last code edit, the turn is blocked once with the exact sentence and the missing step. The agent either runs the check and reports the real output, or rewrites the sentence as unverified. |
| `PreToolUse` (Bash) | before a command runs | A test or build command piped through `tail`/`head` without `set -o pipefail` loses its exit code. The call is denied with the one-line fix. |

If your project already has a `verify` skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.

## How a claim is judged

1. **Scope: one turn.** Everything the agent did between your prompt and its reply.
2. **Last code edit.** `Edit`/`Write` tools, or `sed -i`, `cat > file`, heredocs that write files. Edits to docs, logs, notes, memory, `.env`, temp dirs or `.agent/` state do not count.
3. **Matching check after it.** Tests for *tests pass*; lint/type/build/check tools for *check passed*; render tools for *rendered*; `git push` / `kaggle … push` for *pushed*; tests, checks or renders for *fixed* (a generic script run is not evidence that a fix works).
4. **Its output.** The last summary line decides: `12 passed` backs the claim, `2 failed` contradicts it, `fail 0` is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (`pytest -k`, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ `13 passed`).
5. **Visual fixes.** "Fixed the overlap" followed by a render and a look at the screenshot is `weak`: shown separately, counted neither as verified nor unverified.
6. **Everything invisible → `could not check`**: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
7. **Not a claim:** plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.

Forked sessions are de-duplicated so the same turn is counted once.

## Precision

Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.

| Round | Flagged `unverified` | Author precision | Independent precision | What was fixed next |
|---|---|---|---|---|
| 1 | 38 | 56% | 69% | doc/log/memory edits counted as code edits; versioned `hyperframes@x check` missed |
| 2 | 22 | 53% | 52% | variable paths, `.agent/state.json`, heredoc bodies; judge output by the last summary line |
| 6 | 12 | 12/12 | 11/11 (+1 unsure) | quote-aware multi-line splitting; writing heredocs never count as verification |

Rounds 3–5 were intermediate fixes (heredoc parsing, variable paths, remote-run claims) and are listed in the report. The independent rater is a Claude agent in a fresh context with no access to the author's labels; it is still a model, not a human. A human second rater is the obvious next step, and the labelling sheets are available on request.

**Recall.** Two raters (author + independent agent) then labelled 66 claims the audit had *not* flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. `sober` still prefers `could not check` over a wrong flag, and the full labelling material is described in [`gate/REPORT.md`](gate/REPORT.md).

## What it is not

- Not a judge of *correctness*. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
- Not a leaderboard of models. The per-model table is your sessions only.
- Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
- Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle); other stacks are covered by synthetic tests only, so expect to report a false flag or two. Issues welcome.

## Related work

[claude-guard-hooks](https://github.com/jefftanlearn-eng/claude-guard-hooks) and [groundtruth](https://github.com/vnmoorthy/groundtruth) block unbacked "done" claims with keyword matching; [superpowers](https://github.com/obra/superpowers) asks for verification in a skill (prompt-level). `sober` differs in reading command *output*, scoring your history across agents and models, and refusing to count what it cannot see.

中文说明见 [README.zh-CN.md](README.zh-CN.md)。
