Metadata-Version: 2.5
Name: cognifity-verdict-inspect
Version: 0.1.0a18
Summary: Verdict Inspect — analyze chat exports for drift, hedging, refusals, and quality. Drop in a conversations.json from ChatGPT/Claude/Cursor, get a report.
Project-URL: Homepage, https://cognifity.ai/verdict
Project-URL: Documentation, https://github.com/cognifityai/verdict/tree/main/docs
Project-URL: Repository, https://github.com/cognifityai/verdict
Project-URL: Issues, https://github.com/cognifityai/verdict/issues
Author: Cognifity AI
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: chat-export,drift-detection,evaluation,llm,reporting
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: cognifity-verdict-eval<0.2,>=0.1.0a18
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# Verdict Inspect

PyPI distribution: `cognifity-verdict-inspect`. Command:
`verdict-inspect`.

One-shot drift analysis on a chat export. Drop in a `conversations.json` from
ChatGPT, a Claude.ai export, a supported agent-session JSONL file, or an
OpenAI-format messages dump, and get back a local drift / quality report.

## Why this exists

Continuous LLM observability (the SDK + monitoring path) is the right answer
for production agent traffic. But before a team commits to instrumenting
their stack, they want to know: **does Verdict actually find anything
interesting in my data?**

`verdict-inspect` runs against a file the user already has on their laptop.
Structural and embedding analysis stay local; the optional judge has a separate
privacy boundary described below.

## Usage

In a full Verdict installation, the same one-off workflow is available under
**Evaluate → Inspect JSON**. Paste JSON or choose a file, run the analysis, and
download the result as JSON. The dashboard accepts at most 4 MiB per request,
runs analysis on the Verdict host, does not write the upload or report to the
Verdict store, and keeps semantic analysis and the external judge off until
selected. Enabling the judge also requires an explicit data-egress confirmation.

The command-line interface remains available for larger local exports and
Markdown reports:

```bash
# Auto-detect format
verdict-inspect analyze ~/Downloads/conversations.json

# Force a format
verdict-inspect analyze --format chatgpt ~/Downloads/conversations.json

# Specify report output
verdict-inspect analyze --report ./drift_report.md ~/Downloads/chatlog.jsonl

# JSON output for piping
verdict-inspect analyze --json ~/Downloads/conversations.json | jq .
```

## Supported formats (v0)

- **ChatGPT data export** — the `conversations.json` from Settings → Data Controls → Export
- **Claude.ai data export** — the `conversations.json` from Settings → Account → Export
- **Generic OpenAI messages JSONL** — one JSON object per line, each with `messages: [{role, content}]`
- **Agent-session JSONL** — type-tagged local agent session logs
- **Auto-detect** — looks at file structure and picks a parser

Planned (v1): Cursor `.cursor/chats/`, Gemini Takeout, LangChain message
history files, Llama Index conversation logs.

## What you get back

For a file with enough substantive assistant turns:

1. **Semantic drift** — embedding-distribution shifts across temporal windows
2. **Judge sample** — PASS/FAIL by dimension on stride-sampled turns (requires `ANTHROPIC_API_KEY`)
3. **Structural metrics** — response length, hedge density, refusal rate, apology rate per window

Semantic drift runs key-free. By default it tries
`sentence-transformers/all-MiniLM-L6-v2`, then falls back to the built-in
`HashingEmbedder` if the dependency/model is unavailable. That fallback detects
lexical embedding-distribution changes; it is not a semantic model, and the
report labels it explicitly. Install the local semantic embedder with
`pip install "cognifity-verdict-eval[semantic]"`.

Triggered and non-triggered semantic rows use the same L2-normalized detector
statistics. Each comparison embeds its current and baseline windows once; the
report does not re-embed or independently recompute non-triggered rows.

Turns with fewer than 10 assistant-response words are excluded from windowed
analysis. At least 16 substantive turns are required for a two-window
comparison; 24 create the default early/middle/late split. Each window's judge
sample is capped at 25 turns. Treat small-window output as exploratory rather
than calibrated production evidence. Chat exports do not contain retrieved
context, so the default context-required groundedness dimension is shown as
`n/a` and is not sent to the judge.

## Privacy

Structural metrics and embedding inference run on your machine. The first
MiniLM run may download model weights, but it does not upload the analyzed
conversation. If `ANTHROPIC_API_KEY` is set and the judge is enabled,
`verdict-inspect` sends stride-sampled user and assistant text (up to 4,000
characters each) to the configured Anthropic model. Anthropic credentials and
data-handling terms apply. Pass `--no-judge` or omit the key to keep conversation
content local and receive structural plus embedding analysis only. The v0
inspect judge is Anthropic-only.
