Metadata-Version: 2.5
Name: cognifity-verdict-inspect
Version: 0.1.0a4
Summary: Verdict Inspect — analyze chat exports for drift, hedging, refusals, and quality. Drop in a conversations.json from ChatGPT/Claude/Cursor, get a report.
Project-URL: Homepage, https://cognifity.ai/verdict
Project-URL: Documentation, https://github.com/cognifityai/verdict/tree/main/docs
Project-URL: Repository, https://github.com/cognifityai/verdict
Project-URL: Issues, https://github.com/cognifityai/verdict/issues
Author: Cognifity AI
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: chat-export,drift-detection,evaluation,llm,reporting
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: cognifity-verdict-eval<0.2,>=0.1.0a4
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# Verdict Inspect

PyPI distribution: `cognifity-verdict-inspect`. Command:
`verdict-inspect`.

One-shot drift analysis on a chat export. Drop in a `conversations.json` from
ChatGPT, a Claude.ai export, a supported agent-session JSONL file, or an
OpenAI-format messages dump, and get back a local drift / quality report.

## Why this exists

Continuous LLM observability (the SDK + monitoring path) is the right answer
for production agent traffic. But before a team commits to instrumenting
their stack, they want to know: **does Verdict actually find anything
interesting in my data?**

`verdict-inspect` runs against a file the user already has on their laptop.
Structural and embedding analysis stay local; the optional judge has a separate
privacy boundary described below.

## Usage

```bash
# Auto-detect format
verdict-inspect analyze ~/Downloads/conversations.json

# Force a format
verdict-inspect analyze --format chatgpt ~/Downloads/conversations.json

# Specify report output
verdict-inspect analyze --report ./drift_report.md ~/Downloads/chatlog.jsonl

# JSON output for piping
verdict-inspect analyze --json ~/Downloads/conversations.json | jq .
```

## Supported formats (v0)

- **ChatGPT data export** — the `conversations.json` from Settings → Data Controls → Export
- **Claude.ai data export** — the `conversations.json` from Settings → Account → Export
- **Generic OpenAI messages JSONL** — one JSON object per line, each with `messages: [{role, content}]`
- **Agent-session JSONL** — type-tagged local agent session logs
- **Auto-detect** — looks at file structure and picks a parser

Planned (v1): Cursor `.cursor/chats/`, Gemini Takeout, LangChain message
history files, Llama Index conversation logs.

## What you get back

For a file with enough substantive assistant turns:

1. **Semantic drift** — embedding-distribution shifts across temporal windows
2. **Judge sample** — PASS/FAIL by dimension on stride-sampled turns (requires `ANTHROPIC_API_KEY`)
3. **Structural metrics** — response length, hedge density, refusal rate, apology rate per window

Semantic drift runs key-free. By default it tries
`sentence-transformers/all-MiniLM-L6-v2`, then falls back to the built-in
`HashingEmbedder` if the dependency/model is unavailable. That fallback detects
lexical embedding-distribution changes; it is not a semantic model, and the
report labels it explicitly. Install the local semantic embedder with
`pip install "cognifity-verdict-eval[semantic]"`.

Triggered and non-triggered semantic rows use the same L2-normalized detector
statistics. Each comparison embeds its current and baseline windows once; the
report does not re-embed or independently recompute non-triggered rows.

Turns with fewer than 10 assistant-response words are excluded from windowed
analysis. At least 16 substantive turns are required for a two-window
comparison; 24 create the default early/middle/late split. Each window's judge
sample is capped at 25 turns. Treat small-window output as exploratory rather
than calibrated production evidence.

## Privacy

Structural metrics and embedding inference run on your machine. The first
MiniLM run may download model weights, but it does not upload the analyzed
conversation. If `ANTHROPIC_API_KEY` is set and the judge is enabled,
`verdict-inspect` sends stride-sampled user and assistant text (up to 4,000
characters each) to the configured Anthropic model. Anthropic credentials and
data-handling terms apply. Pass `--no-judge` or omit the key to keep conversation
content local and receive structural plus embedding analysis only. The v0
inspect judge is Anthropic-only.
