Metadata-Version: 2.4
Name: toolgauntlet
Version: 0.1.0
Summary: Deterministic reliability testing for tool-using AI agents
Project-URL: Homepage, https://labs.hilomedia.com/products/toolgauntlet
Project-URL: Documentation, https://github.com/josephnilo/agent-chaos/tree/main/docs
Project-URL: Repository, https://github.com/josephnilo/agent-chaos
Project-URL: Issues, https://github.com/josephnilo/agent-chaos/issues
Project-URL: Changelog, https://github.com/josephnilo/agent-chaos/blob/main/CHANGELOG.md
Author: HiLo Labs
License: MIT
License-File: LICENSE
Keywords: agents,chaos-engineering,evaluation,llm,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.21
Requires-Dist: prometheus-client>=0.20
Requires-Dist: pyyaml>=6.0
Requires-Dist: uvicorn>=0.30
Provides-Extra: dev
Requires-Dist: markdown>=3.6; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: site
Requires-Dist: markdown>=3.6; extra == 'site'
Description-Content-Type: text/markdown

# ToolGauntlet

ToolGauntlet by HiLo Labs is a Python package and CLI for deterministic reliability testing of tool-using AI agents.

## Install

```bash
pip install toolgauntlet
toolgauntlet --version
```

The distribution, primary CLI, and public Python import are all `toolgauntlet`.
The prerelease `agent-chaos` commands and `agent_chaos` import remain available
as compatibility aliases.

## Quick start (CLI)

Generate a complete local baseline with no API key:

```bash
toolgauntlet quickstart --out-dir toolgauntlet-quickstart
```

This writes JSON, Markdown, and HTML baseline reports plus a copyable next-step
regression command. It uses the built-in deterministic adapter with safe logging
enabled and refuses to overwrite its files unless `--force` is supplied.

Build a custom workflow manually:

```bash
toolgauntlet list-suites
toolgauntlet init-suite --id my_suite_v1 --out-dir suites
toolgauntlet run --suite ecommerce_refunds_v1 --runs 10 --out report.json
toolgauntlet report --in report.json --format md --out report.md
toolgauntlet report --in report.json --format html --out report.html
```

Use a saved baseline as a release gate:

```bash
toolgauntlet run --suite ecommerce_refunds_v1 --runs 25 --out baseline.json
toolgauntlet run \
  --suite ecommerce_refunds_v1 \
  --runs 25 \
  --baseline baseline.json \
  --fail-on-regression \
  --regression-tolerance 1.0 \
  --out candidate.json
```

See the copyable CI workflow in [`docs/ci-gate-example.md`](docs/ci-gate-example.md) and a sample report in [`docs/sample-reports/ecommerce_refunds_v1.md`](docs/sample-reports/ecommerce_refunds_v1.md).

Before release, run the same local gate used for MVP validation:

```bash
scripts/release_gate.sh
```

Verify the configured external launch stage (live page now; registries as the
stage advances):

```bash
python scripts/check_launch_readiness.py
```

Signed pack workflow:

```bash
export TOOLGAUNTLET_PACK_KEY=your-shared-signing-key
toolgauntlet sign-pack --suite suites/my_suite_v1
toolgauntlet verify-pack --suite suites/my_suite_v1
toolgauntlet run --suite suites/my_suite_v1 --verify-pack --out report.json
```

OpenTelemetry traces (optional):

```bash
pip install "toolgauntlet[otel]"
# Requires an OTLP/HTTP collector reachable at the default endpoint,
# or pass --otel-endpoint with your collector URL.
toolgauntlet run --suite ecommerce_refunds_v1 --runs 5 --otel-enabled --otel-metrics-enabled
```

## Quick start (Python)

```python
from toolgauntlet import SuiteConfig, run_suite
from toolgauntlet.adapters import deterministic_demo_agent

report = run_suite(
    agent=deterministic_demo_agent,
    suite_path="ecommerce_refunds_v1",
    config=SuiteConfig(runs=10, concurrency=2),
)

report.save_json("report.json")
report.save_markdown("report.md")
```

## OpenAI-compatible adapter

```python
from toolgauntlet import SuiteConfig, run_suite
from toolgauntlet.adapters import OpenAICompatibleAdapterConfig, make_openai_tool_loop_adapter

agent = make_openai_tool_loop_adapter(
    OpenAICompatibleAdapterConfig(
        model="gpt-4o-mini",
        base_url="https://api.openai.com",
    )
)

report = run_suite(
    agent=agent,
    suite_path="ecommerce_refunds_v1",
    config=SuiteConfig(runs=5, concurrency=1),
)
```

## Additional adapter helpers
- `make_async_adapter`: use async agent callables with the synchronous harness interface.
- `make_langchain_like_adapter`: adapt invoke/ainvoke-style runnables.

See examples in [`examples/adapters/`](examples/adapters/).

## Budget-as-Code Proxy

Start and validate:

```bash
toolgauntlet-proxy validate-policy --policy examples/budget_proxy/policy.example.yaml
toolgauntlet-proxy serve \
  --host 127.0.0.1 \
  --port 8080 \
  --policy examples/budget_proxy/policy.example.yaml \
  --ledger logs/usage-ledger.sqlite
```

One-command local stack (proxy + redis + optional otel/prometheus):

```bash
docker compose --env-file .env.proxy -f docker-compose.proxy.yml up --build
```

See full guide: [`docs/compose-deployment.md`](docs/compose-deployment.md)

Build and preview site pages locally:

```bash
pip install -e ".[dev,site]"
toolgauntlet build-site --pages-root site/pages/toolgauntlet --out-dir site/dist/toolgauntlet
python -m http.server 8787 --directory site/dist
```

Key endpoints:
- `GET /healthz`
- `GET /metrics`
- `GET /v1/models`
- `POST /v1/chat/completions`
- `POST /v1/responses`
- `POST /admin/reload-policy`

Common environment variables:
- `OPENAI_API_KEY` or `BUDGET_UPSTREAM_API_KEY`
- `BUDGET_PROXY_API_KEY`
- `BUDGET_POLICY_PATH`
- `BUDGET_AUDIT_LOG_PATH`
- `BUDGET_LEDGER_PATH`
- `BUDGET_PROJECT_HEADER`
- `BUDGET_PROJECT_API_KEY_HEADER`
- `BUDGET_METRIC_NAMESPACE`
- `BUDGET_RATE_LIMIT_BACKEND` (`memory` or `redis`)
- `BUDGET_REDIS_URL` (required when backend is `redis`)
- `BUDGET_REDIS_KEY_PREFIX`
- `BUDGET_OTEL_ENABLED`
- `BUDGET_OTEL_METRICS_ENABLED`
- `BUDGET_OTEL_SERVICE_NAME`
- `BUDGET_OTEL_ENDPOINT`

To enable distributed rate limiting across proxy instances, install Redis support:

```bash
pip install "toolgauntlet[redis]"
```

## Included suites
- `ecommerce_refunds_v1`
- `support_triage_v1`
- `travel_booking_v1`

## Documentation
- docs index: [`docs/README.md`](docs/README.md)
- API reference: [`docs/api-reference.md`](docs/api-reference.md)
- proxy operations: [`docs/proxy-operations.md`](docs/proxy-operations.md)
- compose deployment: [`docs/compose-deployment.md`](docs/compose-deployment.md)
- site preview: [`docs/site-preview.md`](docs/site-preview.md)
- CI gate example: [`docs/ci-gate-example.md`](docs/ci-gate-example.md)
- commercial pack readiness: [`docs/commercial-pack-readiness.md`](docs/commercial-pack-readiness.md)
- customer pack handoff: [`docs/customer-pack-handoff.md`](docs/customer-pack-handoff.md)
- launch operations checklist: [`docs/launch-operations-checklist.md`](docs/launch-operations-checklist.md)
- AI venture operating plan: [`docs/venture-operating-plan.md`](docs/venture-operating-plan.md)
- AI venture operator: [`docs/venture-operator.md`](docs/venture-operator.md)
- release checklist: [`docs/release-checklist.md`](docs/release-checklist.md)
- versioning policy: [`docs/versioning-policy.md`](docs/versioning-policy.md)
- entitlement model: [`docs/entitlement-model.md`](docs/entitlement-model.md)
- support SLA: [`docs/support-sla.md`](docs/support-sla.md)

## Marketing and launch materials
- materials index: [`marketing/README.md`](marketing/README.md)
- website copy: [`site/pages/toolgauntlet/`](site/pages/toolgauntlet)
- pilot sales path: [`site/pages/toolgauntlet/pilot.md`](site/pages/toolgauntlet/pilot.md) and [`docs/pilot-intake.md`](docs/pilot-intake.md)

## Backlog and roadmap
- product backlog: [`TODO.md`](TODO.md)

## Release workflows
GitHub Actions workflows included:
- `.github/workflows/ci.yml`
- `.github/workflows/launch-readiness.yml`
- `.github/workflows/venture-ops.yml`
- `.github/workflows/lead-intake.yml`
- `.github/workflows/release.yml`

Publishing uses trusted publishing via `pypa/gh-action-pypi-publish`.
