Metadata-Version: 2.5
Name: AgentAssure-ai
Version: 0.1.0
Summary: pytest-based QA framework for enterprise AI agents: routing, tools, state changes, permissions, and LLM metrics.
License: Proprietary
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.7
Requires-Dist: pytest>=8.3
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Description-Content-Type: text/markdown

# agentassure

**pytest-based QA for enterprise AI agents.** Talk to your agent in a test, then check what it said, which
tools it called, what it changed in your system of record and whether it respected user permissions.
Tests run live against the agent, combine across repeated runs and personas, and end in a quality gate
your CI can block on.

- **Responses:** exact facts (`response_contains`) and LLM-judged meaning (`response_satisfies`).
- **Behaviour:** routing, tool calls and arguments, multi-agent handoffs, human escalation.
- **Side effects:** before/after diffs of real records; anything the agent changed that the test didn't
  expect fails.
- **Safety:** persona permissions, data exposure, prompt-injection resistance.
- **Quality metrics:** answer relevance, task completion, rubrics, faithfulness and context metrics
  (scored by your own judge LLM).
- **Reports:** console, JSON, JUnit and HTML (with full transcripts), plus optional OpenTelemetry export.

Works with Salesforce Agentforce (install `agentassure-agentforce`), any JSON-over-HTTP agent, or a Python
function.

## Install

```bash
pip install AgentAssure-ai                            # HTTP / Python agents
pip install AgentAssure-ai AgentAssure-ai-agentforce  # Salesforce Agentforce
pip install "AgentAssure-ai[otel]"                    # + OpenTelemetry export
```

Requires Python 3.11+. The `agentassure` command and the pytest plugin are registered on install.

## Quick start

```bash
agentassure init --adapter http --env dev   # agentassure.yaml, env/dev.yaml, .env.example, tests/
agentassure doctor --env dev                # checks config, credentials and agent connectivity
agentassure test --env dev                  # runs the tests (any pytest args work: -k, -n 4, -x, ...)
```

## A first test

```python
import agentassure as aa


@aa.test(agent="service_agent", personas=["west_rep", "east_rep"])
def test_escalation(agent: aa.Conversation, sf: aa.StateProvider, expect: aa.Expect):
    case = sf.create("Case", Subject="Delayed order", Priority="Medium")  # removed after the run

    with aa.step("Ask the agent to escalate"), sf.watch() as diff:
        reply = agent.ask("Escalate case {n}", n=case.CaseNumber)

    with aa.step("Check the reply and what changed"):
        expect(reply).routed_to(topic="Case_Management").tool_called_once("Escalate_Case")
        expect(reply).response_satisfies("Confirms the case is now high priority")
        expect(diff).record(case).changed_only(Priority="High")
        expect(diff).no_unexpected_mutations()
```

- **Fixtures:** `agent` (the conversation), `expect` (soft assertions: every failure is reported, not
  just the first), `state` / `sf` (create, query and diff records) and `row` (for CSV-driven tests).
  The type hints are optional; they turn on autocomplete and hover docs in your editor.
- **`personas=[...]`** runs the test once per test user. Add **`repeat=N`** to run it N times and judge the
  combined results, since agents are non-deterministic.
- **`aa.step(...)`** groups the transcript and checks in the HTML report.

## What you can check

| Area | Examples |
|---|---|
| Response | `response_contains`, `response_not_contains`, `response_satisfies`, `response_max_length` |
| Routing and tools | `routed_to`, `tool_called`, `tool_called_once`, `tool_not_called`, `tool_order`, `no_tool_errors` |
| Multi-agent | `handoff`, `max_handoffs`, `no_handoff_loop`, `escalated_to_human` |
| Permissions and data | `respects_permissions`, `values_not_exposed`, `fields_not_exposed`, `identity_propagated` |
| Security | `attack_blocked`, with ready-made payloads in `aa.attacks` |
| Record changes | `created`, `created_only`, `record(...).changed_only`, `no_unexpected_mutations` |
| LLM metrics | `metric(aa.metrics.ANSWER_RELEVANCE)`, `TASK_COMPLETION`, `RUBRIC`, `faithfulness`, ... |
| Custom | `expect.that(condition, "name")`, `expect.score(...)`, your own `Metric` subclass |

## Data-driven tests

Tests are always Python. To drive one test from a spreadsheet, use
`@aa.cases("cases.csv", unique_id="TestCaseId")`, which runs the test once per row. Add
`sequence_by="Turn"` to group rows into multi-turn conversations.

## Configuration

`agentassure.yaml` defines the agents and run-wide settings. `env/<name>.yaml` holds what differs per
environment: connection, personas (test users) and gates. `${VAR}` and `${VAR:-default}` read from the
environment or a `.env` file.

```yaml
# agentassure.yaml
project: support-agents
agents:
  order_bot:
    adapter: http
    base_url: ${ORDER_BOT_URL:-http://localhost:8080}
    path: /chat
    response: { text: reply, tool_calls: trace.tools }
judge: { provider: openai, model: gpt-4o-mini, api_key: "${AA_JUDGE_API_KEY}" }
metrics:
  answer_relevance: 0.8
```

```yaml
# env/qa.yaml
gates:
  pass_rate: 0.95
  critical_failures: 0
  unexpected_mutations: 0
```

## Command line

| Command | Does |
|---|---|
| `agentassure init` | scaffold a project (`--adapter agentforce \| http`) |
| `agentassure doctor --env qa` | check configuration, credentials, org safety and agent resolution |
| `agentassure test --env qa` | run tests: `--agent`, `--tags "smoke and not slow"`, `--repeat N`, plus any pytest args |
| `agentassure report results.json` | re-render the HTML report |

`agentassure test` exits `0` when the gates pass, `1` when a gate blocks or a test fails, `4` on invalid
configuration and `10` when the target is a production org.

## Extending

Adapters, state providers and metrics are plugins, found through the `agentassure.adapters`,
`agentassure.state_providers` and `agentassure.metrics` entry-point groups. Subclass `AgentAdapter`,
`StateProvider` or `Metric`; each class's docstring lists what to implement. For a one-off metric, decorate
the subclass with `@register_metric` in a `conftest.py`.

## Documentation

The full user guide (configuration, every check, CI setup, troubleshooting) is in `docs/gitbook` in the
source repository.
