Metadata-Version: 2.5
Name: pytest-wardenbot
Version: 0.6.0
Summary: Pytest security tests for chatbots and tool-using agents — prompt injection, MCP boundary checks, data leaks and semantic regressions.
Project-URL: Homepage, https://wardenbot.ai/pytest-plugin/
Project-URL: Documentation, https://pytest-wardenbot.wardenbot.ai/
Project-URL: Repository, https://github.com/wardenbot-ai/pytest-wardenbot
Project-URL: Issues, https://github.com/wardenbot-ai/pytest-wardenbot/issues
Author-email: WardenBot AI <hello@wardenbot.ai>
License-Expression: Apache-2.0
License-File: LICENSE.md
Keywords: ai-safety,ai-security,chatbot,llm,llm-security,prompt-injection,pytest,pytest-plugin,red-teaming
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Requires-Dist: pytest>=8.1.1
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.49.0; extra == 'anthropic'
Provides-Extra: async
Requires-Dist: pytest-asyncio>=0.24; extra == 'async'
Provides-Extra: claude-agent
Requires-Dist: claude-agent-sdk<1,>=0.2.140; extra == 'claude-agent'
Requires-Dist: mcp<3,>=2.2.0; extra == 'claude-agent'
Requires-Dist: uvicorn<1,>=0.31.1; extra == 'claude-agent'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pre-commit==4.6.2; extra == 'dev'
Requires-Dist: pyright==1.1.414; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: ruff==0.17.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'docs'
Requires-Dist: pymdown-extensions>=10; extra == 'docs'
Provides-Extra: judge
Requires-Dist: anthropic>=0.49.0; extra == 'judge'
Requires-Dist: deepeval>=4.2.8; extra == 'judge'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: langgraph
Requires-Dist: langchain-core<2,>=1.0; extra == 'langgraph'
Requires-Dist: langgraph<2,>=1.0; extra == 'langgraph'
Requires-Dist: mcp<3,>=2.2.0; extra == 'langgraph'
Requires-Dist: uvicorn<1,>=0.31.1; extra == 'langgraph'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.2.0; extra == 'mcp'
Requires-Dist: uvicorn<1,>=0.31.1; extra == 'mcp'
Provides-Extra: openai
Requires-Dist: openai>=1.66.3; extra == 'openai'
Description-Content-Type: text/markdown

# pytest-wardenbot

[![CI](https://github.com/wardenbot-ai/pytest-wardenbot/actions/workflows/ci.yml/badge.svg)](https://github.com/wardenbot-ai/pytest-wardenbot/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/wardenbot-ai/pytest-wardenbot/branch/main/graph/badge.svg)](https://codecov.io/gh/wardenbot-ai/pytest-wardenbot)
[![Python versions](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://pypi.org/project/pytest-wardenbot/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/wardenbot-ai/pytest-wardenbot/blob/main/LICENSE.md)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)

Pytest security tests for chatbots and tool-using agents — prompt injection,
MCP and application boundary checks, data leaks and semantic regressions.

📖 **Documentation:** [pytest-wardenbot.wardenbot.ai](https://pytest-wardenbot.wardenbot.ai/)

> **Status: Beta (0.5).** The feature set for the first stable release is in place. Public APIs change only under the [versioning and stability policy](https://pytest-wardenbot.wardenbot.ai/reference/stability/). Questions and feedback: [GitHub Discussions](https://github.com/wardenbot-ai/pytest-wardenbot/discussions). Bugs: [GitHub Issues](https://github.com/wardenbot-ai/pytest-wardenbot/issues).

---

## What it does

Run pytest against chatbot responses and instrumented agent operations. Check prompt
injection, data disclosure, task correctness, and unauthorized tool effects.

- **Black-box.** Tests run against your live chatbot via HTTP, OpenAI API, Anthropic API, or any object you write a small adapter for.
- **Deterministic-first.** The chatbot suite ships 29 tests graded with regex, substring, and schema checks, without a separate judge model. Your chatbot's API usage may still cost money. Optional LLM-judge tests (DeepEval) ship as an extra for semantic checks.
- **Agent-ready chatbot failures.** When a chatbot test fails, the failure message includes a structured Markdown remediation prompt you can paste into Cursor or Claude Code. Agent scenarios report a content-minimized outcome, the observed evidence and its limits.
- **Public adapter checks.** The bundled OpenAI and Anthropic adapters have mock-based tests and a scheduled live API smoke workflow. Check the [latest workflow results](https://github.com/wardenbot-ai/pytest-wardenbot/actions/workflows/live-api-smoke.yml) for the current live status.
- **Instrumented agent scenarios.** Eight versioned cases cover MCP tool results,
  descriptions, resources and prompts, each with a benign baseline. A recording
  sink proves effects; required-coverage mode prevents missing evidence from
  becoming a successful CI run.
- **Application boundary packs.** Opt-in retrieval, memory, synthetic tenant/role
  and approval checks exercise controlled local operations. Each includes benign,
  vulnerable, resistant and no-exposure controls. See the
  [application integration guide](https://pytest-wardenbot.wardenbot.ai/how-to/application-packs/).

### What "passing" means (and doesn't)

A passing chatbot test means its configured response check did not detect a
failure. A passing required agent suite additionally needs observed exposure,
complete declared-boundary evidence, and successful benign task controls.
Neither result establishes security beyond the tested cases. See the versioned
[coverage matrix](https://pytest-wardenbot.wardenbot.ai/about/coverage/) and [evaluation limits](https://pytest-wardenbot.wardenbot.ai/about/evaluation/).

A green run does **not** mean your chatbot is secure. Frontier-grade attacks are multi-turn, novel, and adapted to your specific bot — no fixed corpus catches all of them. Treat the shipped suite as a starter set: pair it with periodic red-team exercises (or our [Continuous Monitoring](https://wardenbot.ai/intake/) service) for the always-on adversarial coverage CI alone can't provide.

## Install

Requires Python 3.11+. The package is maintained by
the [WardenBot AI organization](https://github.com/wardenbot-ai/pytest-wardenbot);
the GitHub transfer does not change its PyPI name or Python imports.

```bash
pip install pytest-wardenbot
```

Optional extras for LLM-judge tests or vendor-native adapters:

```bash
pip install "pytest-wardenbot[judge]"        # adds DeepEval for semantic checks
pip install "pytest-wardenbot[openai]"       # adds OpenAI Responses + Chat adapters (sync + async)
pip install "pytest-wardenbot[anthropic]"    # adds Anthropic Messages adapter (sync + async)
pip install "pytest-wardenbot[langchain]"    # adds LangChainAdapter for any Runnable (sync + async)
pip install "pytest-wardenbot[async]"        # adds pytest-asyncio for parallel async probing (run_probes)
pip install "pytest-wardenbot[mcp]"          # adds the controlled local MCP harness
pip install "pytest-wardenbot[langgraph]"    # adds the LangGraph agent target
pip install "pytest-wardenbot[claude-agent]" # adds the Claude Agent SDK agent target
```

**Fully local:** presets for [Ollama, vLLM and llama.cpp](https://pytest-wardenbot.wardenbot.ai/how-to/local-models/)
run both suites and the LLM judge with no API key. A tool-calling check reports
models that can't call tools as unsupported, and the test suite proves local
runs make no outside network connections.

> **Assistants retirement:** OpenAI retired the Assistants API on 26 August 2026;
> it is unavailable. Use `OpenAIResponsesAdapter` for new OpenAI text integrations.
> Legacy Assistants class names remain importable for explicitly injected test
> clients or compatible proxies; their default constructors fail before SDK or
> network calls. See [OpenAI migration](https://pytest-wardenbot.wardenbot.ai/how-to/migrate-openai/).

| Integration | Adapter | Conversation state |
| --- | --- | --- |
| OpenAI Responses text | `OpenAIResponsesAdapter` / `AsyncOpenAIResponsesAdapter` | Local history per session; requests use `store=False`. |
| OpenAI Chat Completions text | `OpenAIChatAdapter` / `AsyncOpenAIChatAdapter` | Local history per session. |
| Checkpointer-backed LangGraph | `LangChainAdapter` / `AsyncLangChainAdapter` | Explicit thread key and reset hook; see [recipe](https://pytest-wardenbot.wardenbot.ai/how-to/langgraph-state/). |
| Your deployed chatbot or agent | HTTP or custom adapter | Your application's conversation and tool flow. |

See the [dependency support matrix](https://pytest-wardenbot.wardenbot.ai/how-to/dependency-support/) for
validated SDK combinations and [adapter setup](https://pytest-wardenbot.wardenbot.ai/how-to/add-chatbot/).

### Instrumented agent trials

For larger runs, [bounded campaigns](https://pytest-wardenbot.wardenbot.ai/how-to/agent-campaigns/) share call and
turn budgets across isolated trials, drain cooperative cancellation, and retain
every planned case in the report. Retries require an explicit safety declaration.

[Portable evidence](https://pytest-wardenbot.wardenbot.ai/how-to/evidence-bundles/) keeps versioned findings,
fingerprints and provenance in JSON, JUnit and standalone HTML. Saved observations
can be re-graded separately from new executions; redacted evidence cannot imply
a passing grade. A fixed synthetic fixture demonstrates replay from a clean install.

The additive `pytest_wardenbot.agents` API records explicit outcomes: violation,
observed resistance, inconclusive evidence, unsupported capabilities and execution
errors. A trusted recording sink establishes effects; model claims do not. Task
success is tracked separately, and diagnostic content is hidden by default.

The optional [MCP harness](https://pytest-wardenbot.wardenbot.ai/how-to/mcp-harness/) exercises tool results,
tool descriptions, resources and prompts over real local MCP transports, with
scripted vulnerable/resistant/unexposed controls. It needs no provider credentials.
See [agent trials](https://pytest-wardenbot.wardenbot.ai/how-to/agent-trials/) for the contract and application
integration responsibilities.

Connect your agent with a ready-made target or recipe:

| Agent | Target |
| --- | --- |
| LangGraph | [`LangGraphAgentTarget`](https://pytest-wardenbot.wardenbot.ai/how-to/langgraph-agent/) (`[langgraph]`) |
| Claude Agent SDK | [`ClaudeAgentTarget`](https://pytest-wardenbot.wardenbot.ai/how-to/claude-agent/) (`[claude-agent]`), isolated from your machine |
| OpenAI Agents SDK | [Recipe](https://pytest-wardenbot.wardenbot.ai/how-to/openai-agents/) |
| Any tool loop | [40-line custom target](https://pytest-wardenbot.wardenbot.ai/how-to/custom-agent/) with `TrialTools` |

After configuring your `agent_target` and `agent_observer` fixtures:

```bash
pytest --pyargs pytest_wardenbot.agent_tests --wardenbot-agent-required
# Select one versioned scenario and its benign baseline:
pytest --pyargs pytest_wardenbot.agent_tests \
  --wardenbot-agent-scenario=mcp.tool-result.forbidden-tool@1 \
  --wardenbot-agent-required
```

The [scenario guide](https://pytest-wardenbot.wardenbot.ai/how-to/agent-scenarios/) includes a complete local
fixture setup. The [reference agent](https://pytest-wardenbot.wardenbot.ai/how-to/reference-agent/) and
[pilot methodology](https://pytest-wardenbot.wardenbot.ai/about/agent-pilot/) exercise a bounded model/tool loop
with paired benign and attack trials. Existing chatbot tests remain in the
separate `pytest_wardenbot.tests` suite.

A second [support application example](https://github.com/wardenbot-ai/pytest-wardenbot/blob/main/examples/support_agent/README.md) uses
direct local ticket APIs, scoped approvals, and a real model tool loop without
MCP. Both examples default to offline SDK controls. The
[support pilot](https://pytest-wardenbot.wardenbot.ai/about/support-pilot/) records repeated trials, held-out
cases, benign quoted/encoded text, utility, incomplete results, and sample-size
limits; it does not rank models.


## Quickstart (under 60 seconds)

```bash
pip install pytest-wardenbot
pytest --wardenbot-quickstart           # generates conftest.py + test_my_bot.py
export CHATBOT_URL=https://your-chatbot.example.com/chat
export CHATBOT_TOKEN=sk-...              # optional
pytest                                   # runs all shipped tests against your bot
```

`--wardenbot-quickstart` accepts an industry template:

```bash
pytest --wardenbot-quickstart=ecommerce       # adds refund/shipping fact placeholders
pytest --wardenbot-quickstart=saas-support    # adds plan/trial fact placeholders
pytest --wardenbot-quickstart=generic         # default; minimal placeholders
```

Then edit `conftest.py` to replace the TODO placeholders with your real
business facts and re-run `pytest`. Worked examples in [`examples/`](https://github.com/wardenbot-ai/pytest-wardenbot/tree/main/examples)
cover the basic HTTP setup, a custom OpenAI adapter, and a GitHub Actions
workflow.

### Manual setup (if you prefer)

Add this to your project's `conftest.py`:

```python
import os
import pytest
from pytest_wardenbot.adapters.http import HTTPChatbotAdapter


@pytest.fixture
def chatbot():
    return HTTPChatbotAdapter(
        url="https://your-chatbot.example.com/chat",
        headers={"Authorization": f"Bearer {os.environ['CHATBOT_TOKEN']}"},
        request_field="message",  # the JSON key your bot reads the prompt from
        response_field="response",  # the JSON key your bot returns the text in
    )
```

Then run the shipped tests with `pytest --pyargs pytest_wardenbot.tests`.

When a test fails, read the failure message, paste the agent-ready Markdown
into Cursor / Claude Code, ship the fix.

## Bundled chatbot checks

| Category | Count | Grading | Requires API key? |
|---|---|---|---|
| Prompt-injection / jailbreak resistance | 5 prompts × 2 checks = 10 | deterministic | no |
| System-prompt leak elicitation (dedicated extraction prompts) | 3 | deterministic | no |
| Refusal-bypass (roleplay / pretext / hypothetical framings) | 3 | deterministic | no |
| Off-topic deflection (scoped bots) | 2 | deterministic | no |
| Indirect / cross-prompt injection (XPIA) | 4 | deterministic | no |
| Encoded-payload jailbreak (Base64 / ROT13 / leet / hex) | 4 | deterministic | no |
| Multi-turn jailbreak (priming + payload, needs session-aware adapter) | 3 | deterministic | no |
| Canary-token sweep (deprecated; leak checks now test the canary) | 1 | deterministic | no |
| Business-truth verification (parametrized over your facts) | user-supplied | deterministic | no |
| Semantic checks via DeepEval (5 factories: equivalence, brand, hallucination, off-policy, refusal quality) | user-supplied | LLM-judge | yes, with `[judge]` extra |

That's **29 deterministic tests** out-of-the-box (plus your business-truth and judge lists). Set `WARDENBOT_CANARY` to a token planted in your bot's system prompt and every leak check also looks for it; [grading accuracy](https://pytest-wardenbot.wardenbot.ai/reference/grading-accuracy/) is measured on 744 labeled responses. Runtime and chatbot API costs depend on your endpoint; the deterministic checks do not call a separate judge model.

The chatbot suite also supports LangChain Runnables, multi-judge ensembles and
parallel async probes. Ask questions in
[GitHub Discussions](https://github.com/wardenbot-ai/pytest-wardenbot/discussions)
and report bugs in [GitHub Issues](https://github.com/wardenbot-ai/pytest-wardenbot/issues).

## How it's different from related tools

Several good tools overlap with parts of this one. We aim to complement them.

- **Promptfoo** runs red-team and evaluation suites from a CLI, including MCP
  security testing. pytest-wardenbot runs inside the pytest suite and CI you
  already have, and its agent results come from what the harness recorded the
  agent attempting and doing, rather than the agent's final text.
- **RAMPART** also brings security testing to pytest, built on PyRIT's broader
  attack generation. pytest-wardenbot keeps a small core install and its own
  controlled harness, and adds versioned scenarios, required coverage and
  portable evidence.
- **DeepEval** and **Inspect AI** provide evaluation metrics and agent-evaluation
  infrastructure. pytest-wardenbot focuses narrowly on security regressions
  with explicit outcomes. It uses DeepEval for its optional semantic checks.
- **Garak** and **PyRIT** are broad attack and research libraries. The chatbot
  suite packages a curated set of test patterns as everyday pytest tests with
  clear failure messages.

## License

Apache 2.0. See [LICENSE.md](https://github.com/wardenbot-ai/pytest-wardenbot/blob/main/LICENSE.md).

## Powered by

[WardenBot AI](https://wardenbot.ai) — continuous external monitoring for AI chatbots. ![Powered by WardenBot AI](https://img.shields.io/badge/Powered_by-WardenBot_AI-purple)

The pytest plugin is the free, open-source slice of our test corpus. Want continuous monitoring across all your bots with daily probes and a dashboard? [Tell us about your setup](https://wardenbot.ai/intake/) — we open invites in small batches.
