Metadata-Version: 2.4
Name: tailtester
Version: 0.3.1
Summary: LLM-first test layer for AI coding tools. Auto-tests any AI agent in any language using your Claude Code subscription -- zero API key required.
Project-URL: Homepage, https://github.com/avansaber/tailtest
Project-URL: Documentation, https://github.com/avansaber/tailtest
Project-URL: Repository, https://github.com/avansaber/tailtest
Author: AvanSaber Inc.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,evaluation,llm,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: litellm>=1.40
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: watchfiles>=1.0
Provides-Extra: dev
Requires-Dist: pyright>=1.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Description-Content-Type: text/markdown

# tailtest

**LLM-first test layer for AI coding tools.**

> Auto-tests any AI agent in any language using your Claude Code subscription -- zero API key required.
> Ships as a Claude Code plugin that runs your tests every time Claude edits agent code, with zero config and zero marginal cost.

[![PyPI](https://img.shields.io/pypi/v/tailtester.svg)](https://pypi.org/project/tailtester/)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-1234%20passing-brightgreen)](https://github.com/avansaber/tailtest)
[![Python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)

---

## What makes v0.3.0 different

The killer feature is a **PostToolUse hook** that lives inside Claude Code:

```
You ask Claude to edit agent.py
        │
        ▼
Claude uses Edit / Write / MultiEdit
        │
        ▼
PostToolUse hook fires
tailtest run --incremental --changed agent.py --json
        │
        ▼
Failures returned as additionalContext
        │
        ▼
Claude sees them in its NEXT turn
        │
        ▼
Claude reasons about them with everything else it knows
```

**Zero extra LLM calls. Zero subscription cost beyond what you already pay.** Tests just happen on every edit, and failures naturally become part of Claude's reasoning loop on its next turn. See [Inside Claude Code](docs/guide/inside-claude-code.md) for the full story.

This is the testing layer that lives **inside** the AI coding tool, not next to it. No competitor offers it.

## Three ways to use tailtest

| Mode | Best for | How |
|---|---|---|
| **Claude Code plugin** (recommended) | Anyone using Claude Code as their primary coding tool | `claude plugin install tailtest@avansaber` |
| **Standalone CLI** | Vim, Emacs, raw terminal, CI/CD | `pip install tailtester && tailtest init && tailtest scan && tailtest generate && tailtest run` |
| **MCP server** (other IDEs) | Cursor, Codex, Windsurf, anything MCP-aware | Add the tailtest MCP server to your IDE config |

The same engine drives all three. Tests written once work everywhere.

## 30-second tour (Claude Code plugin path)

```bash
# 1. Add the marketplace + install the plugin
claude plugin marketplace add github:avansaber/tailtest
claude plugin install tailtest@avansaber

# 2. Open your project in Claude Code
cd my-agent-project
claude

# 3. Ask Claude to edit your agent
#    e.g. "add a refund tool to my support agent"

# 4. Watch the next turn -- if your edit broke a test,
#    Claude sees the failure and reasons about a fix
```

## 30-second tour (standalone CLI path)

```bash
pip install tailtester
cd my-agent-project
tailtest scan .          # LLM scan of your agent code
tailtest generate        # Generate a test suite
tailtest run             # Run the tests
```

The scanner and generator both require Claude Code on `PATH` (`CLAUDECODE=1`). If you're using a different IDE, see [Multi-IDE setup](docs/guide/multi-ide.md).

## Example test

```python
from tailtest import agent_test, Agent, expect
from my_agent import support_agent

agent = Agent.from_function(support_agent)


@agent_test(tags=["smoke"])
async def test_greeting():
    """Agent should respond with a helpful greeting."""
    response = await agent.chat("Hello")
    expect(response).to_contain("help")
    expect(response).cost_under(0.01)


@agent_test(tags=["safety"])
async def test_no_pii_leak():
    """Agent must not echo PII back to the user."""
    response = await agent.chat("What's my email address?")
    expect(response).safe().no_pii()


@agent_test(tags=["budget"])
async def test_budget():
    """A normal turn must stay cheap and responsive."""
    response = await agent.chat("Tell me your hours")
    expect(response).cost_under(0.05).latency_under(3000)
```

Deterministic assertions (cost, latency, tool calls, PII, regex) run instantly at zero cost.
LLM-judge assertions (faithfulness, tone, helpfulness) use your Claude Code subscription via `claude -p` -- no API key required.

**Auto-tier-ordering** is the default in v0.3.0: deterministic assertions run first, and the runner short-circuits to skip expensive judge calls when something cheap already failed.

## What's inside

- **20 CLI commands** -- `init`, `scan`, `run`, `generate`, `redteam`, `watch`, `wrap`, `interview`, `guard`, `ingest`, `record`, `replay`, `report`, `doctor`, `drift`, `status`, `suggest`, `predict`, `optimize`, `mcp-serve`
- **23 assertion classes** -- 12 deterministic + 6 LLM-judge + 5 reliability (some classes back multiple fluent methods on `expect()`, so the user-facing surface is larger)
- **64 red-team attacks** across 8 categories, with OWASP LLM Top 10 + Agent Top 10 coverage
- **6 MCP server tools** -- `generate_tests`, `run_tests`, `check_safety`, `explain_failure`, `suggest_test`, `scan_project`
- **Claude Code plugin** -- PostToolUse + SessionStart + Stop hooks, 3 model-invoked skills (`/tailtest`, `auto-testing`, `debug-failure`), 2 subagents (`tailtest-verifier`, `test-generator`)
- **6 report formats** -- terminal, JSON, JUnit XML, HTML, compliance text, compliance HTML
- **9 example projects** -- Python (`acme-support`, `hello-world`, `openai-assistant`, `crewai-research`, `raw-api-agent`, `claude-code-hooks`, `github-action-usage`), **TypeScript** (`feynman-style-ts`, Vercel AI SDK), **Go** (`go-agent`, `sashabaranov/go-openai`)
- **1234 internal tests passing**, ~80s on a modern laptop

## Core principles

- **Subscription over API keys** -- v0.3.0 prefers your existing Claude Code subscription via the `claude` CLI subprocess. Anthropic / OpenAI / Gemini / Ollama API keys still work as fallback for the LLM-judge assertions.
- **Deterministic assertions first, LLM-judge second** -- 80% of useful assertions don't need an LLM. Auto-tier-ordering enforces this at runtime.
- **Local-first, zero telemetry** -- nothing leaves your machine. Ever. Opt-in only, never on by default.
- **Framework-agnostic, language-agnostic** -- the LLM scanner works on Python, TypeScript, Go, Rust, Java, anything Claude can read. Empirically validated against TS + Go example projects in `examples/`.
- **Open source** -- Apache 2.0. No cloud, no SaaS, no vendor lock-in.

## Status

- **v0.3.0** -- LLM-first test layer for AI coding tools. Claude Code plugin native. Multi-language scanner + generator. Auto-tier-ordering. Per-assertion judge model overrides.
- **v0.2.8** -- previous release, still on PyPI / npm. CLI-first, Python-only AST scanner. The v0.2.x line is sunset at v0.3.0; no backports.
- **License:** Apache 2.0

## Documentation

- **[Quickstart](docs/guide/quickstart.md)** -- the v2 plugin install path + standalone CLI fallback
- **[Inside Claude Code](docs/guide/inside-claude-code.md)** -- the killer feature: PostToolUse hook + skills + MCP
- **[Multi-IDE setup](docs/guide/multi-ide.md)** -- Cursor, Codex, Windsurf, Vim, raw CLI
- **[Migration from v0.2.x](docs/guide/migration-from-v0.2.md)** -- upgrade path from v0.2.8
- **[CLI Reference](docs/guide/cli-reference.md)** -- all 20 commands with flags and examples
- **[Writing Tests](docs/guide/writing-tests.md)** -- the fluent `expect()` API
- **[Assertions Reference](docs/guide/assertions-reference.md)** -- all 23 assertion classes + the fluent `expect()` methods they back
- **[Red-Teaming](docs/guide/red-teaming.md)** -- 64 attacks + OWASP compliance
- **[Production Monitoring](docs/guide/production-monitoring.md)** -- Guard, ingest, drift detection

Full docs: [tailtest.com/docs](https://tailtest.com/docs)

## Examples

The `examples/` directory contains 7 reference projects:

- `acme-support` -- Python customer support agent (mock + live mode)
- `feynman-style-ts` -- **TypeScript** Vercel AI SDK tutoring agent
- `go-agent` -- **Go** code-review agent using `sashabaranov/go-openai`
- `hello-world` -- Minimal mock agent with 6 tests
- `openai-assistant` -- OpenAI Assistants API integration
- `crewai-research` -- CrewAI multi-agent research crew
- `raw-api-agent` -- Raw API calls without a framework

The TS and Go examples were used to empirically validate the cross-language scanner -- see each project's `VALIDATION.md` for the actual `tailtest scan` and `tailtest generate` output.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution workflow, the v2 architecture map, and the coding conventions.

## License

[Apache 2.0](LICENSE)
