Metadata-Version: 2.5
Name: agentaxiom
Version: 0.1.0
Summary: Evidence-first regression and contract testing for Gemini and Vertex AI agents.
Project-URL: Homepage, https://github.com/Hardonian/AgentAxiom
Project-URL: Repository, https://github.com/Hardonian/AgentAxiom
Project-URL: Issues, https://github.com/Hardonian/AgentAxiom/issues
Project-URL: Changelog, https://github.com/Hardonian/AgentAxiom/blob/main/CHANGELOG.md
Author: AgentAxiom
License: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,gemini,regression-testing,testing,vertex-ai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4.23
Provides-Extra: export
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'export'
Requires-Dist: google-cloud-storage>=2.0; extra == 'export'
Provides-Extra: lint
Requires-Dist: ruff>=0.8; extra == 'lint'
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == 'test'
Provides-Extra: vertex
Requires-Dist: google-genai>=1.0; extra == 'vertex'
Description-Content-Type: text/markdown

# AgentAxiom

Evidence-first regression and contract testing for Gemini and Vertex AI agents.

> Turn an unreliable agent regression into a minimized, replayable, evidence-backed contract violation.

AgentAxiom records a scenario's prompt hash, model/provider identity, region, generation configuration, declared tools, usage, latency, cost, citations and safety metadata. It evaluates deterministic contracts against that record, so CI is offline by default. Live confirmations are opt-in and bounded by a hard call and cost budget.

See the prioritized [100-item go-live gate](docs/GO_LIVE_100.md) for completed repository controls and the required external release approvals.

## MVP capabilities

- JSON Schema output validation and required/forbidden tool-call contracts.
- Grounding/citation, safety action, cost, and latency assertions.
- Offline fixture replay with fail-closed live verification gates.
- Delta-debug conversation minimization for reproducible failing traces.
- Sanitized JSON and HTML evidence bundles.
- Pytest fixture, CLI, and GitHub Actions workflow.
- Deterministic baseline/candidate comparisons with narrowly configurable noise paths.
- Per-tool argument schemas, allowed citation domains, evidence checksums, and Cloud Trace correlation IDs.

## Quick start

```bash
pip install -e .
pytest
agentaxiom check fixture.json examples/contract.json --evidence-dir artifacts/quoted-answer
```

`fixture.json` uses the `Transcript` shape (scenario plus invocation). The evidence bundle emits `evidence.json` and `index.html`; secret-looking values are redacted before writing.

Try the included complete offline path:

```bash
python -m agentaxiom.cli check examples/fixture.json examples/contract.json
python -m agentaxiom.cli compare examples/fixture.json examples/fixture.json
agentaxiom minimize failing.json contract.json --output minimized.json
```

## Live provider boundary

The core deliberately has no implicit network path. Implement the small `AgentClient` protocol for Gemini/Vertex or ADK, then supply `ReplayPolicy(live_verification=True, max_live_calls=N, max_live_cost_usd=X)`. Calls without this explicit opt-in, or beyond either budget, raise `PermissionError`.

## Planned adapters and exports

The core is provider-neutral so Gemini, Vertex AI, ADK, Cloud Trace, Cloud Run, GCS, and BigQuery adapters can preserve each platform's native response metadata without weakening the evidence format. Optional package groups are reserved for the Google provider/export clients:

```bash
pip install -e '.[vertex,export]'
```

`GeminiVertexClient` in `agentaxiom.providers` is the optional recorder adapter; `export_gcs` and `export_bigquery` in `agentaxiom.exports` publish evidence to those sinks. Cloud Run can invoke the CLI or pytest suite directly, and the returned evidence record carries the IDs needed to correlate Cloud Trace spans supplied by the calling service.

## Cloud Run evaluator

The included container exposes an offline-only evaluator, suitable for a CI runner or a protected internal service:

```bash
docker build -t agentaxiom .
docker run -p 8080:8080 agentaxiom
# POST /evaluate with {"transcript": <fixture>, "contract": <contract>}
```

`GET /healthz` returns readiness. `POST /evaluate` never invokes a model; evidence writing is optional and defaults to ephemeral container storage.

### No-code checkup screen

Open the evaluator's root URL to use the built-in plain-language console. It lets non-technical teammates upload a saved run and its rule file, see a simple pass/fail explanation, compare two runs, and download a shareable result. With a local install, start it with:

```bash
agentaxiom serve
# Then open http://localhost:8080
```

The screen uses the same offline evaluator and does not make model calls.

It is designed for a simple hand-off: an engineer records the run and writes the rules once; anyone on the team can then use the browser screen to review the result.

## Regression workflow

1. Capture a sanitized `Transcript` after an agent response, including the exact tools, provider model, region, generation settings and ADK version.
2. Commit it as an offline fixture and pair it with a `Contract`.
3. Run `agentaxiom check` in CI. Failed runs write a self-contained, checksumed evidence bundle.
4. Use `agentaxiom compare baseline.json candidate.json` when validating a model, region, prompt, or dependency upgrade. For a failure whose trigger is independent of deleted context, use `agentaxiom minimize` to delta-debug the trace.
5. Only for a bounded confirmation, create a `ReplayRunner` with `live_verification=True`, a nonzero call limit, and an explicit cost ceiling.

The root [GitHub Action](action.yml) accepts a transcript and contract path for use from another repository.
