Metadata-Version: 2.4
Name: trail-otel
Version: 1.0.0
Summary: Signed OpenTelemetry GenAI spans for AI agents.
Author: Varun Jain
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agents,genai,mcp,observability,opentelemetry,tracing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: cryptography>=42
Requires-Dist: opentelemetry-api>=1.27
Requires-Dist: opentelemetry-exporter-otlp>=1.27
Requires-Dist: opentelemetry-sdk>=1.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: adk
Requires-Dist: google-adk>=1.17; extra == 'adk'
Provides-Extra: dev
Requires-Dist: mcp>=1.28; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: openai>=2.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: mcp>=1.28; extra == 'mcp'
Provides-Extra: openai
Requires-Dist: openai>=2.0; extra == 'openai'
Description-Content-Type: text/markdown

# Trail

[![CI](https://github.com/varmax2511/trail/actions/workflows/ci.yml/badge.svg)](https://github.com/varmax2511/trail/actions/workflows/ci.yml)

**Signed OpenTelemetry GenAI spans for AI agents. Capture, normalize, verify — bring your own backend.**

<!-- Regenerate: asciinema rec trail-demo.cast --overwrite --idle-time-limit 2
     --command "examples/demo_claude_code/run_demo.sh"; then
     agg --theme monokai --speed 1.3 trail-demo.cast docs/trail-demo.gif -->
![Trail demo](https://raw.githubusercontent.com/varmax2511/trail/main/docs/trail-demo.gif)


Trail is a Python SDK that captures what AI agents actually do — every LLM call, tool invocation, MCP call, and skill execution — as OpenTelemetry spans with a small Trail extension namespace. It signs each session with Ed25519 and exports via OTLP to any OTel backend (Grafana, Honeycomb, Datadog, Chronosphere, ...). Trail does not store, query, or dashboard. Storage and query are your existing backend's job.

## Why Trail

When an agent misbehaves in production, three questions are surprisingly hard to answer:

- **Which tool ran, in what order, with what inputs?** Existing tracers are LLM-call-shaped, not agent-shaped.
- **Was that MCP server response trying to inject instructions?** No mainstream tracer flags this.
- **Is this skill the same code it was yesterday?** Skill substitution leaves no trace by default.

Trail adds the three things that are missing: an **agent-aware tool taxonomy** (`internal` / `mcp` / `skill` / `builtin`), **MCP injection flagging** on tool responses, and a **skill hash** that detects silent substitution — all as standard OpenTelemetry spans, so any OTel backend ingests them with no translation layer.

## How Trail answers them

Trail models an agent run as an OpenTelemetry **span tree** — one `invoke_agent` root span per session, with every LLM call, tool, MCP call, and skill nested underneath — and layers a `trail.*` attribute namespace on top. That structure, plus three purpose-built attributes, is what turns each question above into a query.

**Which tool ran, in what order, with what inputs?**
Every tool invocation becomes an `execute_tool` span tagged with `gen_ai.tool.name` and `trail.tool_type` (`internal` / `mcp` / `skill` / `builtin`) — the agent-shaped distinction a plain LLM tracer never draws. Order and nesting come from the OpenTelemetry SDK's `contextvars` propagation, which stays correct across `async`/`await` and concurrent `asyncio` tasks, so each span attaches to the right parent. Inputs and outputs are recorded as `trail.input_hash` / `trail.output_hash` (SHA-256, computed off the hot path) plus a sensitivity flag — tamper-evident identity of the payloads without storing the payloads themselves.

```
gen_ai.operation.name = "execute_tool"
gen_ai.tool.name      = "get_customer_record"
trail.tool_type       = "mcp"
trail.input_hash      = "sha256:..."
```

**Was that MCP server response trying to inject instructions?**
When Trail wraps an MCP `call_tool`, it runs the *response* through a YAML injection ruleset — instruction-override, system-prompt injection, role override, credential-exfil phrasing (override via `TRAIL_MCP_RULES`) — and stamps the span with `trail.mcp.injection_flag`. A response that says "ignore your previous instructions and…" lands as an ordinary span with `trail.mcp.injection_flag = true`, next to `trail.mcp.server_id` for provenance.

**Is this skill the same code it was yesterday?**
`wrap_skill()` records `trail.skill.hash` — a SHA-256 over the skill's source (`trail.skill.hash_method = "source"`, with a `qualname-fallback` for C-extensions and lambdas). Same skill → same hash; a silent swap → a different hash on today's span versus yesterday's. Diff the attribute across two sessions and substitution is visible.

**Then you ask where you already look.** Trail only captures — the questions get *answered* in your backend. In dev mode that's the session JSONL (`~/.trail/sessions/{trace_id}.jsonl`), and `trail verify-export` proves none of it was altered after the fact (and pinpoints the span if it was). In prod, it's an ordinary attribute filter — `trail.tool_type = "mcp" AND trail.mcp.injection_flag = true` — in Grafana, Honeycomb, or Datadog.

## Quickstart — OpenAI agent (60 seconds)

```bash
pip install 'trail-otel[openai]'
```

```python
import openai
import trail

trail.auto_instrument()           # detects openai, instruments it

with trail.session(agent_id="content-pipeline"):
    client = openai.OpenAI()
    client.chat.completions.create(model="gpt-4o", messages=[...])
```

That's it. By default Trail writes spans to `~/.trail/sessions/{trace_id}.jsonl` and a short summary to stderr. Zero infrastructure.

To ship to your OTel backend instead:

```bash
export TRAIL_EXPORT=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector:4317
```

Async OpenAI (`AsyncOpenAI`) is instrumented automatically by the same `trail.auto_instrument()` call.

## Quickstart — Claude Code

Trail ships a `trail-hook` console script. Wire it into `~/.claude/settings.json`. A single binary handles all three events — it reads the event name from Claude Code's stdin payload and dispatches internally.

```json
{
  "hooks": {
    "PreToolUse": [
      { "matcher": "*", "hooks": [{ "type": "command", "command": "trail-hook" }] }
    ],
    "PostToolUse": [
      { "matcher": "*", "hooks": [{ "type": "command", "command": "trail-hook" }] }
    ],
    "SessionEnd": [
      { "hooks": [{ "type": "command", "command": "trail-hook" }] }
    ]
  }
}
```

**Already have hooks?** Claude Code's `hooks.<EventName>` is an array — Trail composes alongside whatever is already there. Append a new `{matcher, hooks}` block per event rather than replacing the array. Trail runs sequentially with your existing hooks and never blocks them (it exits 0 even on internal errors).

**See it end to end:** `examples/demo_claude_code/run_demo.sh` replays a real Claude Code session against `trail-hook` (no infrastructure, no API key) — captures the tool taxonomy, flags a prompt-injection riding in on an MCP-fetched GitHub issue, then `verify-export` proves the session and catches a tamper.

Every Claude Code tool call — including MCP calls and skills — is now captured. The `SessionEnd` hook is the moment the session gets its Merkle root + Ed25519 signature. (Signing is wired to `SessionEnd`, which fires once when the session terminates — **not** `Stop`, which fires at the end of every turn and would leave later turns' spans unsigned.)

## Quickstart — Google ADK

Google's Agent Development Kit is OpenTelemetry-native, so Trail rides ADK's own
`execute_tool` spans rather than re-instrumenting — one `auto_instrument()` call
adds the tool taxonomy and MCP injection flag ADK doesn't produce, and wrapping
the run in `trail.session()` signs it.

```python
import trail
from google.adk.runners import Runner

trail.auto_instrument()           # detects google.adk, enriches its tool spans

with trail.session(agent_id="support-triage", provider="gcp.vertex"):
    runner.run(user_id="u1", session_id="s1", new_message=msg)
```

Every ADK tool call now carries `trail.tool_type` (`McpTool` → `mcp`,
ADK-provided search/memory tools → `builtin`, your `FunctionTool`s → `internal`)
and MCP responses are scanned for injection (`trail.mcp.injection_flag`).

**Try it in dev mode first — zero infrastructure.** Dev mode is the default, so
the two lines above already write every ADK span to
`~/.trail/sessions/{trace_id}.jsonl` locally (no network). Run your agent, then
inspect what was captured:

```bash
cat ~/.trail/sessions/<trace_id>.jsonl | jq .      # spans + trail.tool_type
```

**Signing is opt-in.** With no keys present, sessions are simply *unsigned* —
the minimal setup: spans + `trail.tool_type` + MCP injection flag, no
tamper-evidence, no signing overhead. Turn it on when you want it:

```bash
trail generate-keys                                # once; enables signing
trail verify-export ~/.trail/sessions/<trace_id>.jsonl
# → VALID  (N spans, signature valid, key fpr ...)
```

> **Dev-mode note:** don't also enable ADK's own Cloud Trace / OTel exporter
> while running dev mode. Trail configures the tracer provider; if ADK sets one
> first, Trail's local JSONL won't attach. Just add the two Trail lines and
> leave ADK's own tracing off.

When it looks right locally, ship the *same* code to your backend — ADK exports
OTLP, so set `TRAIL_EXPORT=otlp` and `OTEL_EXPORTER_OTLP_ENDPOINT` (e.g.
Chronosphere) and the spans flow there instead. See
[`docs/backends/chronosphere.md`](docs/backends/chronosphere.md), and
[`examples/adk_manual_instrumentation.py`](examples/adk_manual_instrumentation.py)
for the framework-agnostic manual path (no adapter required).

> ADK has no first-class "skill", so skill-hashing stays with `trail.wrap_skill`,
> which composes with ADK. Parallel/merged tool calls are a documented v1 gap.

## What you get on each span

Standard OpenTelemetry GenAI attributes:

```
gen_ai.operation.name      = "chat" | "execute_tool" | "invoke_agent"
gen_ai.provider.name       = "openai" | "anthropic"
gen_ai.request.model       = "gpt-4o"
gen_ai.tool.name           = "get_customer_record"
gen_ai.usage.input_tokens  = 1240
```

Plus the Trail extension — the novel part:

```
trail.tool_type            = "internal" | "mcp" | "skill" | "builtin"
trail.mcp.server_id        = "acme-crm-mcp"
trail.mcp.injection_flag   = false
trail.skill.hash           = "sha256:..."
trail.input_hash           = "sha256:..."
trail.output_hash          = "sha256:..."
```

In Grafana or Honeycomb, these render as ordinary GenAI spans. The `trail.*` attributes are queryable like any other attribute (`trail.tool_type = "mcp" AND trail.mcp.injection_flag = true`).

## Metrics? Use the Collector's `spanmetrics` connector

Trail emits **spans only** — no Prometheus scrape endpoint and no OTel metrics. To get rate / error / duration counters or a "MCP injections per minute" panel, drop the OpenTelemetry Collector's `spanmetrics` connector into your pipeline and label by `trail.tool_type`, `trail.mcp.injection_flag`, etc. Span backends (Tempo's metrics-generator, Datadog APM metrics, Honeycomb derived columns) offer equivalent backend-side derivations. Two signal types at the source would duplicate the signal — the Collector composes them cleanly.

## Verifying a session offline

Each session is signed once at session end with Ed25519 over a Merkle root of its span content. Anyone with the public key can verify it later — no Trail infrastructure required:

```bash
trail verify-export session.jsonl
# → VALID  (132 spans, signed 2026-06-06T10:02:14Z, key fpr sha256:abcd...)
```

Modified spans, removed spans, and added spans are all detected by the Merkle root mismatch.

Generate a keypair:

```bash
trail generate-keys
# → ~/.trail/keys/trail.key  (private, chmod 600)
# → ~/.trail/keys/trail.pub  (public)
```

## Dev mode vs prod mode

| Mode | Storage | Signing | Network | Use it for |
|------|---------|---------|---------|------------|
| Dev (default) | `~/.trail/sessions/*.jsonl` + stderr summary | Off | None | Local debugging |
| Prod | OTLP to your backend | On (Ed25519 + Merkle, at session end) | OTLP | Shipping to Grafana / Honeycomb / Datadog / Chronosphere |

Per-backend setup (endpoint, auth, query examples) lives in
[`docs/backends/`](docs/backends/README.md) — Grafana Tempo, Honeycomb, Datadog,
Chronosphere. For clusters, see [`docs/deployment/kubernetes.md`](docs/deployment/kubernetes.md).

## v1 scope, honestly

**In:** OpenAI SDK adapter, Google ADK adapter, Claude Code hooks, OTel GenAI emission, tool taxonomy, MCP injection flagging, skill hash, session-checkpoint signing, OTLP transport, dev-mode JSONL, `verify-export`, `generate-keys`.

**Not yet:** LangChain / LlamaIndex / AutoGen adapters, HTTP proxy, sidecar deployment, CloudTrail / CloudWatch transports, encrypted sensitive-content side-store, GDPR erasure workflow, multi-org config, KMS-backed signing.

**Known v1 limitations:**

- In-process capture is **suppressible** by the agent code. Trail v1 is positioned as a developer debugging tool. Suppression-resistant capture (proxy / sidecar) is a v2 theme.
- A process crash before session end leaves spans **unsigned** (still exported, just unverifiable). Per-event signing is v2.
- Claude Code hooks expose tool events, not LLM calls — so the LLM-token detail you'd get from the OpenAI adapter is absent from the Claude Code path. Tool taxonomy, MCP flagging, and skill hash come through on both paths.

## Roadmap (v2 themes)

Suppression-resistant capture (HTTP proxy + sidecar), per-event / checkpoint signing for crash safety, additional framework adapters (LangChain, LlamaIndex, AutoGen), encrypted sensitive-content side-store, GDPR erasure workflow, KMS-backed signing, additional transports (CloudTrail, CloudWatch).

## Project structure

See `trail_hld.md` for the high-level design and `CLAUDE.md` for implementation conventions.

## Security

Trail produces signed, tamper-evident telemetry — reports against the signing /
verification path are taken seriously. See [`SECURITY.md`](SECURITY.md) for the
disclosure process and what is in scope.

## License

Apache-2.0. See `LICENSE` and `NOTICE`.
