Metadata-Version: 2.4
Name: verifierlab
Version: 0.2.0rc2
Summary: Verifier Assurance Lab — local-first campaigns for verifier robustness evaluation
Project-URL: Homepage, https://github.com/fraware/verifierlab
Project-URL: Documentation, https://fraware.github.io/verifierlab/
Project-URL: Repository, https://github.com/fraware/verifierlab
Project-URL: Issues, https://github.com/fraware/verifierlab/issues
Project-URL: Changelog, https://github.com/fraware/verifierlab/blob/main/CHANGELOG.md
Author: VerifierLab Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: assurance,evaluation,robustness,verifier
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.11
Requires-Dist: cryptography<47,>=42
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: pydantic-settings<3,>=2.3
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: rich<15,>=13.7
Requires-Dist: typer<1,>=0.12
Provides-Extra: adapters
Requires-Dist: gymnasium<1.3,>=0.29; extra == 'adapters'
Requires-Dist: harbor<0.21,>=0.17; (python_version >= '3.12') and extra == 'adapters'
Requires-Dist: inspect-ai<0.4,>=0.3.200; extra == 'adapters'
Requires-Dist: openenv<0.5,>=0.4; extra == 'adapters'
Provides-Extra: dashboard
Provides-Extra: dev
Requires-Dist: hypothesis<7,>=6.100; extra == 'dev'
Requires-Dist: moto[s3]<6,>=5.0; extra == 'dev'
Requires-Dist: mypy<2,>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio<1,>=0.23; extra == 'dev'
Requires-Dist: pytest-cov<7,>=5.0; extra == 'dev'
Requires-Dist: pytest<9,>=8.2; extra == 'dev'
Requires-Dist: ruff<1,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml<7,>=6.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mike<3,>=2.1; extra == 'docs'
Requires-Dist: mkdocs-material<10,>=9.5; extra == 'docs'
Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
Provides-Extra: envassure
Provides-Extra: gym
Requires-Dist: gymnasium<1.3,>=0.29; extra == 'gym'
Provides-Extra: harbor
Requires-Dist: harbor<0.21,>=0.17; (python_version >= '3.12') and extra == 'harbor'
Provides-Extra: inspect
Requires-Dist: inspect-ai<0.4,>=0.3.200; extra == 'inspect'
Provides-Extra: kubernetes
Requires-Dist: kubernetes<35,>=29.0.0; extra == 'kubernetes'
Provides-Extra: nemo
Provides-Extra: objectstore
Requires-Dist: boto3<2,>=1.34; extra == 'objectstore'
Provides-Extra: openenv
Requires-Dist: openenv<0.5,>=0.4; extra == 'openenv'
Provides-Extra: rl
Provides-Extra: rllib
Requires-Dist: ray[rllib]==2.48.0; extra == 'rllib'
Provides-Extra: sandbox
Provides-Extra: slurm
Provides-Extra: trainer
Description-Content-Type: text/markdown

<div align="center">

```
 __     _______ ____  ___ _____ ___ _____ ____  _        _    ____  
 \ \   / / ____|  _ \|_ _|  ___|_ _| ____|  _ \| |      / \  | __ ) 
  \ \ / /|  _| | |_) || || |_   | ||  _| | |_) | |     / _ \ |  _ \ 
   \ V / | |___|  _ < | ||  _|  | || |___|  _ <| |___ / ___ \| |_) |
    \_/  |_____|_| \_\___|_|   |___|_____|_| \_\_____/_/   \_\____/ 
```

### Verifier Assurance Lab

**Stress-test graders, reward functions, and policy checkers under optimization — not another model leaderboard.**

[![CI](https://github.com/fraware/verifierlab/actions/workflows/ci.yml/badge.svg)](https://github.com/fraware/verifierlab/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue.svg)](pyproject.toml)
[![Status](https://img.shields.io/badge/status-0.2.0rc2-blue.svg)](CHANGELOG.md)

`verifierlab` · CLI: `valab`

</div>

---

## Pitch

**VerifierLab** is a local-first lab for running reproducible campaigns that ask a hard question: *when something is optimized against your verifier, does the verifier still do its job?*

You wrap a grader or reward function, run ordinary baselines alongside optimized-*tagged* attack cohorts under declared budgets and access models, then rebuild reports from content-addressed artifacts after **freeze → adjudicate → release-labels**. Finding no exploit is evidence under those conditions — not a proof of correctness. This release candidate (`0.2.0rc2`) meets executable RC gates 1–6 in `tests/test_acceptance_gates.py`; do not claim soundness or SOTA verifier assurance. Honest gaps (default process-local trust boundary, research-grade attack depth) are listed in [docs/limitations.md](docs/limitations.md).

## Why it exists

Modern agents and training loops lean on **verifiers**: graders, reward models, rubrics, policy checkers. Those components can be gamed — reward hacking, rubric gaming, eval cheating — and the failure modes only show up under pressure.

VerifierLab exists so you can:

- Separate **ordinary** behavior from **optimized-tagged** attack cohorts
- Keep runs **reproducible** with content-addressed artifacts and digests
- Stay **local-first**: the base install has no PyTorch, Ray, Kubernetes, or model SDKs
- Record **budgets** (queries, steps, wall time) instead of pretending attacks are free

## Status

**Release candidate** (`0.2.0rc2`). Gates 1–6 are encoded and passing in `tests/test_acceptance_gates.py` (isolation, optimization, measurement, repair, reproduction, developer lifecycle). Core loop — workspace, process workers, typed decisions, broker metering with worker budget vouchers, vault v2, freeze→adjudicate→release, StatsPlan reports, packs A–F, and the offline tutorial — is implemented and tested. See [docs/limitations.md](docs/limitations.md) and [SECURITY.md](SECURITY.md). Optional adapters and launchers ship behind extras or explicit dry-run modes. APIs and CLI may still shift before 0.2.0.

## Install

Requires Python `>=3.11,<3.14`. [uv](https://docs.astral.sh/uv/) is recommended.

```bash
git clone https://github.com/fraware/verifierlab.git
cd verifierlab
uv sync --extra dev
# or: pip install -e ".[dev]"

uv run valab doctor
uv run valab --version
```

From a published package (when available on your index):

```bash
pip install verifierlab
valab doctor
```

## Quick start

Offline refund tutorial (full lifecycle):

```bash
uv run valab init
uv run valab inspect examples.refunds.verifier:grade
uv run valab run campaigns/refund-blackbox.yaml
uv run valab campaign freeze .valab/runs/<run-id>
uv run valab campaign adjudicate .valab/runs/<run-id> --campaign campaigns/refund-blackbox.yaml
uv run valab campaign release-labels .valab/runs/<run-id>
uv run valab report builds .valab/runs/<run-id>
```

Faster smoke:

```bash
uv run valab campaign validate campaigns/fake-smoke.yaml
uv run valab campaign run campaigns/fake-smoke.yaml --processes
```

## What you can do

| Goal | How |
| ---- | --- |
| **Wrap a verifier** | Decorate a grader with `@verifier`, then `valab inspect MODULE:ATTR` |
| **Run a campaign** | `valab run PATH` or `valab campaign run PATH` (attack plane only) |
| **Freeze, adjudicate, release** | `freeze` → `adjudicate` → `release-labels` before reports |
| **Rebuild a report** | `valab report builds RUN_DIR` → HTML / JSON / CSV (post-release) |
| **Compare a repair** | Call `compare_repair` in Python: regression corpus plus a mandatory fresh attacker |
| **Plan sample size** | `valab stats power` for binomial power / *n* planning |

Full CLI table: [docs/cli.md](docs/cli.md).

## Optional extras

Keep the base install light. Pull only what you need:

| Extra | Use |
| ----- | --- |
| `gym` | Gymnasium adapter |
| `inspect` | Inspect AI adapter |
| `harbor` | Harbor adapter (Python ≥3.12) |
| `openenv` | OpenEnv adapter |
| `nemo` | NeMo Gym HTTP client surface |
| `objectstore` | S3-compatible object store |
| `kubernetes` | Kubernetes launcher client |
| `adapters` | Bundle of the adapter extras above |

```bash
uv sync --extra gym
# or: pip install -e ".[gym]"
```

Details: [docs/adapters.md](docs/adapters.md).

## Documentation

| Doc | Topic |
| --- | ----- |
| [Getting started](docs/getting-started.md) | Install and tutorial |
| [Concepts](docs/concepts.md) | Verifiers, cohorts, store, access models |
| [Architecture](docs/architecture.md) | Engine, store, run bundles |
| [Methodology](docs/methodology.md) | How to interpret campaign results |
| [Attacks](docs/attacks.md) | Built-in strategies |
| [Labels and statistics](docs/labels-and-stats.md) | Vault, freeze, error rates |
| [Threat model](docs/threat-model.md) | Assets, boundaries, non-goals |
| [Limitations](docs/limitations.md) | What not to claim |
| [Security policy](SECURITY.md) | Vulnerability reporting |

## Contributing

Contributions are welcome — new attack strategies, adapters, campaigns, docs, and bug fixes all help.

- Read [CONTRIBUTING.md](CONTRIBUTING.md) for setup, PR expectations, and coding standards
- Browse [open issues](https://github.com/fraware/verifierlab/issues), especially ones labeled for newcomers
- Keep the **base** dependency graph light; heavy stacks belong behind extras
- Follow the [Code of Conduct](CODE_OF_CONDUCT.md)

```bash
uv sync --extra dev
uv run ruff check src tests
uv run mypy
uv run pytest
uv run python scripts/check_base_imports.py
```

## Security

Do not open public issues for security-sensitive reports. Prefer a private [GitHub Security Advisory](https://github.com/fraware/verifierlab/security/advisories/new), or see [SECURITY.md](SECURITY.md).

## License

[Apache-2.0](LICENSE).

## Links

- Repository: https://github.com/fraware/verifierlab
- Issues: https://github.com/fraware/verifierlab/issues
- Changelog: [CHANGELOG.md](CHANGELOG.md)
