Metadata-Version: 2.4
Name: pisama-verifier-gym
Version: 0.1.0
Summary: Verifier audit harnesses, datasheets, and agreement metrics from Pisama
Project-URL: Homepage, https://pisama.ai
Project-URL: Documentation, https://github.com/Pisama-AI/pisama/tree/main/packages/pisama-verifier-gym
Project-URL: Repository, https://github.com/Pisama-AI/pisama
Project-URL: Issues, https://github.com/Pisama-AI/pisama/issues
Project-URL: Changelog, https://github.com/Pisama-AI/pisama/blob/main/packages/pisama-verifier-gym/CHANGELOG.md
Author-email: Pisama Team <team@pisama.ai>
License: MIT
License-File: LICENSE
Keywords: agents,agreement,ai,datasheets,evals,llm-judges,verifiers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == 'dev'
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Description-Content-Type: text/markdown

# pisama-verifier-gym

[![PyPI version](https://img.shields.io/pypi/v/pisama-verifier-gym.svg)](https://pypi.org/project/pisama-verifier-gym/)
[![Python versions](https://img.shields.io/pypi/pyversions/pisama-verifier-gym.svg)](https://pypi.org/project/pisama-verifier-gym/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![CI](https://github.com/Pisama-AI/pisama/actions/workflows/ci.yml/badge.svg)](https://github.com/Pisama-AI/pisama/actions/workflows/ci.yml)

Audit harnesses for verifiers: LLM judges, reward functions, graders, and failure
detectors. The package ships a verifier datasheet template, a worked WildChat
derailment judge datasheet, sanitized backing artifacts, and a no-dependency
agreement calculator.

## Install

```bash
pip install pisama-verifier-gym
```

## Quick Start

```bash
pisama-verifier-gym agreement
pisama-verifier-gym validate artifact.json
pisama-verifier-gym gate baseline.json candidate.json
pisama-verifier-gym render artifact.json --output datasheet.md
```

```python
from pisama_verifier_gym import agreement_table, load_builtin_verdicts

rows = load_builtin_verdicts()
for stat in agreement_table(rows):
    print(stat.vendor_a, stat.vendor_b, stat.raw_agreement, stat.positive_specific_agreement)
```

## What Is Included

- `TEMPLATE.md`: the verifier datasheet template.
- `datasheets/derailment-wildchat.md`: a filled worked example for a task
  derailment LLM judge.
- `data/wildchat_v3_derailment_verdicts.jsonl`: sanitized per-trace panel
  verdicts, joinable to WildChat by `source_trace_id`.
- `data/contested_adjudication.sanitized.json`: contested-label adjudication
  record with lineage fields and conversation text removed.
- `data/judge_agreement.json`: aggregate agreement artifact from the same lane.

Conversation text is not redistributed. WildChat is distributed by AI2 under
its own license terms.

## Python API

```python
from pathlib import Path
from pisama_verifier_gym import (
    agreement_table,
    load_verdict_rows,
    pairwise_agreement,
    verdict_balance,
)

rows = load_verdict_rows(Path("verdicts.jsonl"))
table = agreement_table(rows)
balance = verdict_balance(rows)

anthropic_google = pairwise_agreement(
    rows,
    "claude-sonnet-4-6",
    "gemini-2.5-flash-lite",
)
```

Each pair reports:

- usable row count after pairwise abstention drops
- raw agreement
- positive specific agreement
- Cohen's kappa

Read raw packaged assets:

```python
from pisama_verifier_gym import read_datasheet, read_template

print(read_template())
print(read_datasheet("derailment-wildchat"))
```

## CLI

```bash
# Built-in WildChat derailment verdicts
pisama-verifier-gym agreement

# A custom JSONL export with the same per_vendor_verdicts shape
pisama-verifier-gym agreement path/to/verdicts.jsonl

# Machine-readable output
pisama-verifier-gym agreement --json

# Validate the Verifier Gym contract
pisama-verifier-gym validate artifact.json

# Compare two artifacts with the same verifier ids
pisama-verifier-gym compare baseline.json candidate.json --json

# Fail on F1 drops, PSA collapse, abstention spikes, threshold drift, or
# unexpected fingerprint changes
pisama-verifier-gym gate baseline.json candidate.json

# Render machine-generated datasheet tables as Markdown
pisama-verifier-gym render artifact.json --output datasheet.md

# Export Pisama calibration reports into the gym contract
pisama-verifier-gym export-calibration backend/data/calibration_report.json \
  --llm-report backend/data/llm_detector_calibration.json \
  --output backend/data/verifier_gym/current.json \
  --positive-manifest backend/data/verifier_gym/positive_rich_manifest.json
```

The validator fails hard when a verifier lacks rubric lineage, a dataset
fingerprint, an input visibility policy, a lane policy, or when synthetic data
can feed published metrics. The gate is intentionally fingerprint-aware: by
default a candidate must be compared against the previous run for the same
dataset fingerprint.

## Why This Exists

High raw agreement is not enough when the positive class is rare. The included
WildChat example shows raw agreement of 0.96 to 0.98, while positive specific
agreement is 0.00 on the same slice. That distinction decides whether a
published verifier metric is useful or misleading.

## Development

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
ruff check src tests
mypy src/pisama_verifier_gym
pytest -q
python -m build
```

Internal Pisama development keeps checked-in gym artifacts under
`backend/data/verifier_gym/`. Regenerate them from real calibration outputs with
`python backend/scripts/verifier_gym/export_current.py`, then validate, render,
and gate before promoting a new baseline.

## License

Code, documentation, and sanitized artifacts in this package are MIT licensed.
