Metadata-Version: 2.4
Name: agent-log-verifier
Version: 0.3.1
Summary: Post-hoc log-analysis dashboard for AI agent transcripts (Claude Code JSONL). Reads session logs to spot order-prompt / tool-call mismatches; a log-reading tool for a human maintainer, not an anomaly monitor.
Project-URL: Repository, https://github.com/skova-digital/agent-log-verifier
Project-URL: Changelog, https://github.com/skova-digital/agent-log-verifier/blob/master/CHANGELOG.md
Project-URL: Issues, https://github.com/skova-digital/agent-log-verifier/issues
Author: Skova Digital
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: pydantic>=2.0
Provides-Extra: cli-rich
Requires-Dist: rich>=13.0; extra == 'cli-rich'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# agent-log-verifier

[![CI](https://github.com/skova-digital/agent-log-verifier/actions/workflows/ci.yml/badge.svg)](https://github.com/skova-digital/agent-log-verifier/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/agent-log-verifier)](https://pypi.org/project/agent-log-verifier/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](./LICENSE)

Post-hoc log-analysis dashboard for AI agent transcripts. Reads Claude
Code's own JSONL session logs and helps you spot cases where an order
prompt and the resulting tool-call history do not match up.

> **What this is not.** This is not a security tool, not an anomaly
> monitor, not a violation catcher, and not a threat-hunting product.
> It reads the same transcript a human maintainer would read, and lays
> the parts of it that are easy to miss (which paths were touched,
> which subagents replied, whether a save was cross-checked afterward)
> into a report you can skim in place of re-reading the JSONL by hand.
> Names like `COMPLIANCE-001` and `INCIDENT-001` in the code and CLI
> output are internal category labels for log-event lookups, not
> compliance-audit or security-incident classifiers.

`alv scan <project-dir>` reads Claude Code's JSONL transcripts and
returns a report noting, after the fact, whether the actions an order
prompt asked for actually appear in the transcript that followed --
without touching the agent, the project it worked on, or any running
system. It only writes to a report file when you ask it to.

![Dashboard rendered from the synthetic demo project: an amber stop-suspicion banner, one stopped-suspected subagent and four auto-verified completions](./docs/images/dashboard-demo.png)

## Why

Agentic coding tools speak on behalf of themselves. "Saved," "verified,"
"done" are all lines the agent itself types into its own final message,
alongside everything else it produced. Most of the time those lines are
accurate. Sometimes they are not: a `Write` call happens, but the
follow-up cross-check that was supposed to confirm the file actually
landed never runs, and the closing report goes out anyway. Or the order
prompt says "look into this only, do not implement," and a stray `Write`
slips in under time pressure regardless.

Catching this by hand means re-reading the transcript, or re-running
`Glob`/`Read` yourself against whatever paths the prompt mentioned, or
rebuilding which of a session's dozens of tool calls actually touched
the file in question. That is mechanical, transcript-shaped work --
exactly the kind a small, plain-Python tool can take over.

`agent-log-verifier` does that specific piece: given an order prompt
and the transcript that followed it, it pulls out the constraints the
prompt actually stated (a "do not X" line, a target save path, a
follow-up confirmation step) and cross-checks the tool-call history
against them. It also answers a second, unrelated but equally
mechanical question -- "is this subagent still running, or did it
quietly stop?" -- from the same transcripts, without asking the agent
or its host system anything.

Notes are candidates for a human to read, not a final ruling.
Natural-language order prompts are ambiguous enough that fully
automated judgement is not this tool's goal (see [Known
Limitations](#known-limitations) below for what that actually costs in
practice).

## Install

Requires Python >= 3.12.

```bash
pip install agent-log-verifier
```

This registers the `alv` command. To work from source instead:

```bash
git clone https://github.com/skova-digital/agent-log-verifier
cd agent-log-verifier
pip install -e .[dev]
```

## Quickstart

Every example below runs against a synthetic, made-up project (a blog
site redesign) built by a script in this repo -- no real transcript,
no real project, no confidential data of any kind.

```bash
python scripts/gen_demo_logs.py
alv scan demo/
```

`scripts/gen_demo_logs.py` writes a small Claude-Code-shaped project
directory to `demo/` (main-session JSONL files plus a `subagents/`
subdirectory per session, mirroring the real Claude Code project
layout). It contains four scenarios, each shaped to be picked up by
exactly one of this tool's lookups:

| Scenario | What happens | Lookup it demonstrates |
|---|---|---|
| `sess-normal-compliant` | Agent writes a file, then confirms it exists via `Glob` | COMPLIANCE-001, 遵守 (followed) |
| `sess-illusory-save` | Agent writes a file, then reports "saved" without ever cross-checking | COMPLIANCE-001, 逸脱疑い (suspected deviation) |
| `sess-prohibited-write` | Prompt says "do not implement at this stage"; agent writes anyway | COMPLIANCE-002, 逸脱疑い |
| `sess-liveness-demo` | One subagent's delegation never gets a matching completion signal; a second one does | LIVENESS-001, `stopped_suspected` vs `completed` |

Scanning a single subagent transcript in isolation makes the contrast
between the matched and unmatched save scenarios easiest to read. The
save-without-confirm one (`sess-illusory-save`) is this tool's
headline case:

```
$ alv scan demo/sess-illusory-save/subagents/agent-illusory.jsonl
agent-log-verifier v0.3.0 -- scan result
target: demo/sess-illusory-save/subagents/agent-illusory.jsonl
sessions found: 1 (main: 1, subagents: 0)

[COMPLIANCE]
  [WARNING] judgement: 逸脱疑い  /tmp/blog-demo/article-list-spec.md への保存後、Glob/Read による実在確認が見つからない
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/article-list-spec.md への保存を確認

summary: 1 warning, 0 critical, 1 info
```

The two lines are both correct and both about the same path, at two
different levels of granularity: the file genuinely was written
(COMPLIANCE-003, 遵守), but the order prompt's "confirm existence
before reporting" step was never carried out (COMPLIANCE-001, 逸脱
疑い). Compare against the followed scenario, where the same two
cross-checks both come back clean:

```
$ alv scan demo/sess-normal-compliant/subagents/agent-normal.jsonl
agent-log-verifier v0.3.0 -- scan result
target: demo/sess-normal-compliant/subagents/agent-normal.jsonl
sessions found: 1 (main: 1, subagents: 0)

[COMPLIANCE]
  [INFO] judgement: 遵守  /tmp/blog-demo/plan.md は保存後に実在確認済み（遵守）
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/plan.md への保存を確認

summary: 0 warning, 0 critical, 2 info
```

Scanning the whole `demo/` directory at once runs every scenario
through every lookup, including subagent status, and mixes in the
main-session view (which sees each subagent's `Task` delegation as
its own, separate transcript with no `Write` calls of its own -- see
[How it works](#how-it-works) for why that produces additional,
independently correct notes about the same paths):

```
$ alv scan demo/
agent-log-verifier v0.3.0 -- scan result
target: demo/
sessions found: 9 (main: 4, subagents: 5)

[COMPLIANCE]
  [WARNING] judgement: 逸脱疑い  成果物パス /tmp/blog-demo/article-list-spec.md への保存が見つからない
  [WARNING] judgement: 逸脱疑い  成果物パス /tmp/blog-demo/plan.md への保存が見つからない
  [INFO] judgement: 遵守  禁止事項「この段階では実装しないこと」は遵守されている
  [WARNING] judgement: 逸脱疑い  /tmp/blog-demo/article-list-spec.md への保存後、Glob/Read による実在確認が見つからない
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/article-list-spec.md への保存を確認
  [INFO] judgement: 遵守  /tmp/blog-demo/plan.md は保存後に実在確認済み（遵守）
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/plan.md への保存を確認
  [WARNING] judgement: 逸脱疑い  禁止事項「この段階では実装しないこと」に反する可能性のあるツール呼び出しが見つかった（1件）

[LIVENESS]
  [INFO] judgement: completed  COMPLETED agent-illusory (writer)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [INFO] judgement: completed  COMPLETED agent-completed (planner)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [WARNING] judgement: stopped_suspected  STOPPED? agent-stopped-suspect (researcher)  最終更新: <N>分前  完了報告・完了通知・正常終了の形いずれもログに無し
  [INFO] judgement: completed  COMPLETED agent-normal (writer)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [INFO] judgement: completed  COMPLETED agent-prohibited (engineer)  最終更新: <N>分前  完了報告の受領をログで自動確認

summary: 5 warning, 0 critical, 8 info
```

(`<N>分前` above stands in for the minutes elapsed since each demo
event -- `gen_demo_logs.py` stamps everything relative to its own run
time, with `agent-stopped-suspect`'s last event backdated 45 minutes,
so the numbers start small and grow until you regenerate the demo;
the status estimate is a function of wall-clock time at scan time,
not a fixed constant.)

`--format json` and `--format markdown` are also available, for
feeding into other tooling or writing a saved report file:

```bash
alv scan demo/ --format json
alv scan demo/ --format markdown --output report.md
```

`alv watch <project-dir>` polls the same subagent-status check on an
interval instead of running once (`Ctrl+C` to stop):

```bash
alv watch demo/ --poll-interval 10
```

Two more subcommands render scan results as HTML instead of terminal
text, for anyone who would rather look at a browser tab than read
JSON:

`alv dashboard <project-dir>` runs one scan and writes a single,
self-contained HTML file (all CSS/JS inlined, zero external requests)
to `./alv-dashboard.html`, then opens it in the default browser:

```bash
alv dashboard demo/
alv dashboard demo/ --output report.html --no-open
```

`alv serve <project-dir>` starts a local HTTP server (bound to
`127.0.0.1` only) that re-scans on an interval (default 10s) and
serves a live-updating dashboard -- the page polls the server in the
background and refreshes in place, with a "最終更新 N秒前" (last
updated N seconds ago) label and a small status dot so it is visible
at a glance that the loop is actually still running, not just sitting
on stale data:

```bash
alv serve demo/ --port 8799 --interval 10
```

`scripts/agent-monitor.bat` is a double-click launcher for `alv
serve` against this repo's own Claude Code project log directory --
see [Live dashboard on Windows](#live-dashboard-on-windows) below.

See `alv scan --help` / `alv report --help` / `alv watch --help` /
`alv dashboard --help` / `alv serve --help` for the full option list
(`--detector`, `--stale-minutes`, `--stopped-minutes`,
`--no-include-subagents`).

### Live dashboard on Windows

`scripts/agent-monitor.bat` starts `alv serve` using this repo's own
`.venv`, so it can be double-clicked (or run from a Desktop shortcut)
without opening a terminal. With no argument it follows the most
recently modified project directory under
`%USERPROFILE%\.claude\projects` (Claude Code's per-project transcript
layout); pass a directory as the first argument -- e.g. in a
shortcut's "Target" field -- to pin one. Right-click the file, choose
"Create shortcut", and drag the shortcut to the Desktop; the `.venv`
referenced must already exist (`python -m venv .venv && pip install -e
.` from this repo's root). Closing the console window it opens (or
`Ctrl+C` inside it) stops the loop.

## What it looks for

Four families of lookups plus one analysis view, all read-only and all
post-hoc:

| Rule | What it reports |
|---|---|
| COMPLIANCE-001 | A save the order prompt asked to cross-check (Glob/Read after Write) that was never cross-checked |
| COMPLIANCE-002 | A tool call that appears to conflict with a stated prohibition (「実装しないこと」 and the like) |
| COMPLIANCE-003 | An ordered artifact path with no matching Write/Edit anywhere in the transcript -- or, on the 遵守 side, the confirmed save |
| COMPLIANCE-004 | A "tests pass" claim with no test-run command in the preceding tool calls |
| COMPLIANCE-005 | A fresh TODO/FIXME/NotImplemented marker added to a code file, followed by a completion claim that does not disclose it |
| COMPLIANCE-006 | Changed files falling outside every project the order prompt mentioned (scope-drift candidates; path-structural, stated as such in the finding text) |
| LOOP-001..004 | Consecutive error streaks; identical retries that never once succeed; no-progress identical repeats; retries that ignore an explicit user correction |
| INCIDENT-001 | Executed data-loss commands (`rm -rf`, `git reset --hard`, `DROP TABLE`, ...) surfaced as facts -- whether they were authorized is the operator's call |
| LIVENESS-001 | Subagent status (running / completed / stopped_suspected / unknown), with zero-token auto-verification of completions from the parent transcript |

The dashboard adds an error-breakdown tab (内訳) that classifies every
failed tool_result into 10 seeded categories (pre-read edit, missing
path, permission-hook block, ...) plus an honest その他 bucket for
anything the dictionary does not recognize yet.

Two standing rules shape everything in that table. Lookups are
error/result-anchored: repetition, volume, and activity are normal
agent behavior, so a rule that could flag a healthy session's busiest
pattern is treated as wrong by design -- only failure and non-progress
are worth a row. And every rule was shipped against evidence from real
transcript mining, including the rules that evidence killed (a
spawn-surge lookup was rejected because it would have flagged the
healthiest sessions on this machine).

## How it works

```
Claude Code (looked at, not touched)
        |  writes
        v
Transcript JSONL files (on disk)
        |  read-only
        v
parser/claude_code.py   -- JSONL -> NormalizedEvent (pydantic),
                            tolerant of unknown fields/types; a
                            malformed line becomes a ParseWarning
                            and is skipped, never halting the whole
                            scan
        v
extraction/constraints.py -- order-prompt text -> constraint
                            candidates (do-not / artifact-path /
                            scope), via keyword + local-window
                            heuristics, not a full NLP parse
        v
lookups/  -- compliance (COMPLIANCE-001..004) / completeness
             (COMPLIANCE-005) / scope (COMPLIANCE-006) /
             loops (LOOP-001..004) / incident (INCIDENT-001) /
             liveness (LIVENESS-001, 4-value status)
             (class names in code retain the `Detector` suffix for
             API compatibility -- these are log-event lookups, not
             anomaly detectors)
        v
report/  -- cli_output.py / json_output.py / markdown_output.py /
             html_output.py (static + live dashboard) /
             breakdown.py (failed-tool_result categorization)
```

A few design points worth knowing before reading a report:

**Two separate transcripts, two separate opinions about the same
path.** A main session's `Task` tool_use only carries the delegated
prompt text (`input.prompt`) -- it has no `Write` calls of its own,
even when the subagent it spawned wrote a file. Scanning a whole
project directory therefore runs order-prompt cross-checks against
the main-session transcript AND each subagent transcript
independently, and a COMPLIANCE-003 "no write found" note from the
main-session's point of view can sit right next to a COMPLIANCE-003
"write confirmed" note from the subagent's point of view for the
same path -- both are correct, at their own transcript's level of
visibility. See the [full `demo/` scan](#quickstart) above for
exactly this pattern.

**Path normalization handles Windows/Git-Bash form mismatches.**
`F:\dev\project\file.md` and `/f/dev/project/file.md` refer to the
same file but do not look alike as strings; `normalize_path()` folds
both into one comparable form (drive letter uppercased, separators
unified, case-insensitive comparison) before any path is compared
against a `Write`/`Glob`/`Read` call's actual `input`. Getting this
wrong silently breaks path matching rather than raising -- it was
one of the concrete bugs found during development (see Known
Limitations).

**Extraction windows, not a full parse.** Constraint extraction
looks for keywords ("しない", "禁止", a confirmation phrase like
"Globで実在確認") and then only scans a fixed character radius
around them for a path or a governing verb, rather than running a
real Japanese morphological parse. This keeps the dependency
footprint at just `pydantic`, but it is also the direct cause of the
over-matching history documented below -- a keyword-window
heuristic finds coincidental proximity, not grammar.

## Known Limitations

v0.1's compliance lookup (save-without-confirm/COMPLIANCE-001 only)
produced a high over-matching rate against real-world Claude Code
sessions: 607 of 611 findings were flagged as suspected deviations
(the one-by-one classification behind that figure is what drove the
v0.2 redesign). v0.2 shipped the improvements v0.1's README had deferred
(a confirmation-keyword-proximity window for path extraction,
excluding reference-list-style mentions, and an early-return for
Write/Edit-zero sessions) plus two new rules (COMPLIANCE-002/003)
and main-session turn segmentation. A subsequent real-log re-scan
and code review found and fixed several further over-matching /
under-matching patterns (prompt-wide "do-not" suppression hiding
unrelated required saves, `.tsx`/`.jsx` extension truncation, an
inline reference-marker fallback erasing the whole prompt,
`CLAUDE.md`-style reference-mention noise). Order-prompt cross-check
notes remain candidates for human review, not a final ruling --
fully automated judgement is not a goal of this release either.

**Turn-straddling is a known blind spot.** Constraints stated in one
user turn and acted on several turns later can fall outside the
per-turn cross-check window; the residue of real-log scans is a mix
of this limitation and genuine candidates, and the tool does not
currently distinguish the two.

**Constraint extraction is a candidate list, not a ruling.** The
3-category extractor (do-not / artifact-path / scope) is
deliberately keyword- and proximity-based rather than a full natural
-language parse, so double negation, sarcasm, or a "do not" and its
scope written across two unrelated sentences can all still confuse
it. Every COMPLIANCE-002/003 note is meant to be read as "worth a
second look," not "confirmed mismatch."

**Single first-party adapter.** Only Claude Code's JSONL transcript
format is parsed. Other agent frameworks emit different log shapes
entirely and are out of scope for this release.

## Known tech debt

`src/agent_log_verifier/detectors/compliance.py` has grown past 900
lines as COMPLIANCE-002/003/004 and their over-matching-reduction
fixes were added on top of v0.1's COMPLIANCE-001 logic (newer rules
went into their own modules: `completeness.py`, `scope.py`,
`incident.py`, `loops.py`). Splitting it into a
`detectors/compliance/` package (e.g. separate modules for path
extraction/normalization, the save-without-confirm check, and the
artifact-path checks) is deferred to a later release rather than
done as part of a review-fix pass, to avoid mixing a structural
refactor with behavior-fixing commits. (The `detectors/` directory
name is kept for import compatibility -- these are log-event lookups
by purpose, not anomaly detectors.)

## License

[MIT](./LICENSE)
