Metadata-Version: 2.4
Name: tracely-ai
Version: 0.2.0
Summary: Tracely Python SDK: automatic + manual agent tracing, exported to Tracely over OTLP.
Project-URL: Homepage, https://github.com/Jwuthri/Tracely
Project-URL: Documentation, https://github.com/Jwuthri/Tracely/blob/master/sdk/README.md
Project-URL: Source, https://github.com/Jwuthri/Tracely
Project-URL: Issues, https://github.com/Jwuthri/Tracely/issues
License-Expression: MIT
License-File: LICENSE
Keywords: agents,evals,llm,observability,opentelemetry,tracing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.25
Requires-Dist: opentelemetry-sdk>=1.25
Provides-Extra: all
Requires-Dist: anthropic; extra == 'all'
Requires-Dist: claude-agent-sdk; extra == 'all'
Requires-Dist: crewai; extra == 'all'
Requires-Dist: google-adk; extra == 'all'
Requires-Dist: google-genai>=0.3; extra == 'all'
Requires-Dist: langchain-openrouter>=0.1.0; extra == 'all'
Requires-Dist: litellm>=1.40; extra == 'all'
Requires-Dist: llama-index; extra == 'all'
Requires-Dist: mistralai>=1.0; extra == 'all'
Requires-Dist: openai-agents; extra == 'all'
Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-claude-agent-sdk>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-crewai>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-google-adk>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-google-genai>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-groq>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-mistralai>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-openai-agents>=0.1.0; extra == 'all'
Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic; extra == 'anthropic'
Requires-Dist: openinference-instrumentation-anthropic>=0.1.0; extra == 'anthropic'
Provides-Extra: bedrock
Requires-Dist: openinference-instrumentation-bedrock>=0.1.0; extra == 'bedrock'
Provides-Extra: claude-agent-sdk
Requires-Dist: claude-agent-sdk; extra == 'claude-agent-sdk'
Requires-Dist: openinference-instrumentation-claude-agent-sdk>=0.1.0; extra == 'claude-agent-sdk'
Provides-Extra: crewai
Requires-Dist: crewai; extra == 'crewai'
Requires-Dist: openinference-instrumentation-crewai>=0.1.0; extra == 'crewai'
Provides-Extra: google
Requires-Dist: google-genai>=0.3; extra == 'google'
Requires-Dist: openinference-instrumentation-google-genai>=0.1.0; extra == 'google'
Provides-Extra: google-adk
Requires-Dist: google-adk; extra == 'google-adk'
Requires-Dist: openinference-instrumentation-google-adk>=0.1.0; extra == 'google-adk'
Provides-Extra: google-genai
Requires-Dist: google-genai>=0.3; extra == 'google-genai'
Provides-Extra: groq
Requires-Dist: openinference-instrumentation-groq>=0.1.0; extra == 'groq'
Provides-Extra: langchain
Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'langchain'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40; extra == 'litellm'
Provides-Extra: llama-index
Requires-Dist: llama-index; extra == 'llama-index'
Requires-Dist: openinference-instrumentation-llama-index>=0.1.0; extra == 'llama-index'
Provides-Extra: mistral
Requires-Dist: mistralai>=1.0; extra == 'mistral'
Requires-Dist: openinference-instrumentation-mistralai>=0.1.0; extra == 'mistral'
Provides-Extra: mistral-sdk
Requires-Dist: mistralai>=1.0; extra == 'mistral-sdk'
Provides-Extra: openai
Requires-Dist: openinference-instrumentation-openai>=0.1.0; extra == 'openai'
Provides-Extra: openai-agents
Requires-Dist: openai-agents; extra == 'openai-agents'
Requires-Dist: openinference-instrumentation-openai-agents>=0.1.0; extra == 'openai-agents'
Provides-Extra: openllmetry
Requires-Dist: opentelemetry-instrumentation-anthropic>=0.30b0; extra == 'openllmetry'
Requires-Dist: opentelemetry-instrumentation-openai-v2>=2.0b0; extra == 'openllmetry'
Provides-Extra: openrouter
Requires-Dist: langchain-openrouter>=0.1.0; extra == 'openrouter'
Provides-Extra: openrouter-openai
Requires-Dist: openai>=1.0; extra == 'openrouter-openai'
Provides-Extra: xai
Requires-Dist: openai>=1.0; extra == 'xai'
Description-Content-Type: text/markdown

# `tracely-ai` — instrument agents + the CI gate CLI

Two things in one small package:

1. an **instrumentation SDK** — `tracely.init()` activates the OpenAI/Anthropic/LangChain/LiteLLM auto-instrumentors so your existing code is traced with **zero span code** (the default path); a custom `SpanProcessor` stamps the active `tracely.trace()` run context onto every span — including the zero-touch provider spans. Manual context managers remain the escape hatch. Everything emits standard `gen_ai.*` / OpenInference attributes **plus** Tracely's first-class `tracely.*` hints, so the backend can populate its agent-semantic columns; and
2. the **`tracely` CLI** — `tracely simulate`, `tracely gate` and `tracely replay`, which turn an agent's behaviour into a pull-request check (exit 0/1 + GitHub status/comment).

The core depends only on `opentelemetry-sdk` + the OTLP HTTP exporter (Python ≥ 3.10); a provider extra adds that provider's auto-instrumentor.

```bash
pip install "tracely-ai[openai]"   # provider extras: [openai] [anthropic] [langchain] [litellm] [all]
# or: pip install tracely-ai       # core only (manual API + CLI)
# import name is `tracely_sdk`; the CLI is `tracely` (entry point tracely_sdk.cli:main)
```

> Already using OpenTelemetry / OpenInference / LangGraph instrumentation? You don't need the instrumentation half — point your existing OTLP exporter at `POST {endpoint}/v1/traces` with `Authorization: Bearer <ingest-key>` and set the `tracely.*` attributes below. This SDK is just the ergonomic path. The **CLI**, however, is how you wire Tracely into CI.

---

## 1. Instrument an agent

### Automatic (the default — zero span code)

`init()` activates the auto-instrumentors; `trace()` attaches the run context; `@observe` adds
function-level spans. No manual span code.

```python
import tracely_sdk as tracely
from openai import OpenAI

tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
             service_name="support-agent", env="prod", instrument="auto")

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    OpenAI().chat.completions.create(model="gpt-4o", messages=[...])   # GENERATION span, captured

@tracely.observe(as_type="agent")            # args→input, return→output, auto-nested
def plan(goal): ...
```

`instrument` is `"auto"` (every importable provider SDK), an explicit list (`["openai",
"anthropic"]`), or `False`. The `tracely.trace()` hints flow onto **every** span inside it — including
the provider spans the instrumentor creates — via a custom `SpanProcessor`. Streaming token usage
needs `stream_options={"include_usage": True}`.

### Serving an agent Tracely drives (`traceparent`)

If Tracely calls your agent — `tracely simulate`, a scenario gate, the Emulate tab — it **mints the
trace id**, sends it as a W3C `traceparent` header, and POSTs its conversation id in the body under
the endpoint's `session_key`. Honour both and your real trajectory lands inside the turn:

```python
@app.post("/api/chat")
def chat(body: dict, request: Request):
    with tracely.trace(
        agent="support-agent",
        conversation=body.get("conversation_id"),          # the endpoint's session_key
        traceparent=request.headers.get("traceparent"),    # joins Tracely's turn span
    ):
        return run_agent(body["message"])
```

Skip it and the turn is Tracely's own span alone — request in, reply out. Your tool calls still get
traced, but onto a **separate** trace with a conversation id of your own, so nothing correlates and
every step/tool-level evaluator has nothing to grade. A header that carries no usable span is logged
and ignored, never fatal. Already running OTel server instrumentation (FastAPI/Starlette/Flask)? The
incoming context is ambient — pass `conversation` and leave `traceparent` unset.

**Also covered:** LangChain/LangGraph (`[langchain]` — graphs nest, node names become steps), LiteLLM
(`instrument=["litellm"]` — 100+ providers via one callback), and a non-patching drop-in
(`from tracely_sdk.openai import OpenAI` / `wrap_openai`). Under `"auto"`, when the LangChain
instrumentor is present it owns LLM spans and the provider instrumentors are skipped to avoid
duplicate spans (override with an explicit list). Full guide: the docs [Automatic instrumentation](https://github.com/Jwuthri/Tracely/blob/master/docs/pages/automatic.mdx) page.

### Manual / custom spans (the escape hatch)

For anything the auto path doesn't cover. Each `with` block is a span; nesting builds the tree.

```python
import tracely_sdk as tracely

tracely.init(endpoint="http://localhost:8000", api_key="tracely_dev_key",
             service_name="support-agent", env="prod")

with tracely.agent("support-agent", version="v3", conversation="conv-1", turn=0) as a:  # AGENT span = run root
    tracely.set_io(a, input=user_msg, output=answer)                 # what the agent received / returned
    with tracely.thinking(agent="support-agent") as th:             # THINKING span (reasoning)
        tracely.set_io(th, output=reasoning); tracely.set_usage(th, thinking_tokens=120)
    with tracely.llm("gpt-4o", agent="support-agent") as g:          # GENERATION span
        tracely.set_io(g, input=messages, output=completion)
        tracely.set_usage(g, input_tokens=812, output_tokens=96)
    with tracely.tool("get_order", agent="support-agent") as t:      # TOOL span
        try:
            result = get_order(order_id)
            tracely.set_io(t, input={"order_id": order_id}, output=result)
        except Exception as e:
            tracely.error(t, str(e))                                 # level=ERROR → the failure signal

tracely.flush()   # force-flush the exporter (call before the process exits)
```

### Span context managers — one per observation type
| Call | Span type | Sets |
|---|---|---|
| `agent(slug, *, version, run_id, role, conversation, turn, user, trace_name, handoff_from, edge="delegate")` | `AGENT` (run root) | `tracely.agent.id`/`.version`/`.run_id`/`.role`, `tracely.conversation.id` + `session.id`, `tracely.turn.index`, `tracely.env`; `user`→`tracely.user.id`, `trace_name`→`tracely.trace.name`; `handoff_from`→ a handoff edge (`caller`→this agent, `edge.type`). |
| `llm(model, *, agent, temperature, top_p, max_tokens, frequency_penalty, presence_penalty, seed, tool_calls, metadata)` | `GENERATION` | `gen_ai.request.model` + the sampling params as `gen_ai.request.*`; `tool_calls`→`tracely.tool_calls` (tools the model **requested**); `metadata`→`tracely.metadata.*`. |
| `tool(name, *, agent)` | `TOOL` | `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`. |
| `thinking(name="thinking", *, agent, model)` | `THINKING` | reasoning emitted as its own span; optional `model`. |
| `retriever(name="retrieve", *, agent)` | `RETRIEVER` | a retrieval step — query in `set_io(input=)`, hits in `set_io(output=)`. |
| `embedding(model, *, agent)` | `EMBEDDING` | `gen_ai.request.model`; record tokens with `set_usage(input_tokens=)`. |
| `guardrail(name="guardrail", *, agent)` | `GUARDRAIL` | a safety/policy check — verdict in `set_io(output={"action": "allow"\|"block"})`. |
| `chain(name, *, agent)` | `CHAIN` | a grouping span (e.g. a RAG pipeline) — nest other spans inside it. |
| `turn(turn_id, *, index)` / `step(name, *, step_id)` | marker / generic | `tracely.turn.*` / `tracely.step.*`. |

### Annotating spans
- `set_io(span, *, input=None, output=None)` → `tracely.input` / `tracely.output` (objects are JSON-encoded; message content is a `{role, content:[blocks]}` object or a content-block list).
- `set_usage(span, *, input_tokens=None, output_tokens=None, thinking_tokens=None)` → `gen_ai.usage.input_tokens` / `output_tokens` / `reasoning_tokens`.
- `set_metadata(span, **kv)` → `tracely.metadata.<key>` — arbitrary tags (e.g. prompt version, tenant), surfaced in the span's Metadata and searchable.
- `set_state(delta, span=None, *, max_bytes=4096)` → `tracely.state.<channel>` — see [Shared state](#shared-state-langgraph-state-scratchpads-stores) below.
- `error(span, message="")` → marks the span `StatusCode.ERROR` (→ `level=ERROR` in Tracely) — this is *the* failure-detection signal.
- `flush()` → force-flush the OTLP exporter.

### What Tracely reads
Standard `gen_ai.*` / OpenInference attributes, plus first-class hints that become **indexed columns** on the span row: `tracely.agent.id`/`.version`/`.role`, `tracely.user.id`, `tracely.trace.name`, `tracely.conversation.id`, `tracely.turn.id`/`.index`, `tracely.step.id`/`.name`, `tracely.observation.type`, `tracely.tool_calls`, `tracely.handoff.*` + `tracely.edge.type`, the `gen_ai.request.*` sampling params, and `tracely.env` (`prod|staging|ci|dev` — the gating axis). Agent slug + version are auto-registered into the Postgres registry on ingest.

### Declaring a conversation's agent catalog
Pass `agents=[{name, description, tools:[...]}]` to `tracely.trace(...)` (or call `set_agents([...])`) to declare the agents/tools a conversation uses — emitted once as `tracely.agents`. The backend stores this per conversation; the UI's **Conversation Agents** panel renders it (with per-tool run counts) and the LLM judge can read it as `@LIST_AGENT`. Without it, Tracely still derives the agent view from the spans.

### Shared state (LangGraph `State`, scratchpads, stores)

Harnesses thread one mutable object through the graph. Tracely records the **delta** — the channels a step wrote — never a snapshot: what you debug is *"`plan` was emptied at step 4, in `replan`"*, and re-stamping the whole state on every span bloats the trace for no gain. The **State** drawer on a conversation folds the deltas back into the full object and shows the per-step diff.

**LangGraph needs no code.** A node's return value *is* its channel update, and the LangChain instrumentor already captures it as the span's output; ingest marks those spans `tracely.state_source=output` so the UI reads them as deltas. Just `init(instrument=["langchain"])`.

For state your harness manages itself, call `set_state`. **Scope is the span you attach it to** — the level is already on the span, so there's nothing extra to declare:

```python
with tracely.step("planner") as s:
    tracely.set_state({"plan": plan, "retries": 0})   # this step
```

Each channel becomes its own `tracely.state.<channel>` attribute (not one blob), so a query can read a single channel; values over `max_bytes` are truncated per key so one fat channel can't crowd out the small ones. Within a conversation a "store" is just state that outlives a turn — attach it to the trace's root span instead of a step span.

> The implicit span is the OTel *current* span, which exists inside `step()`/`tool()`/`llm()` but **not** inside an auto-instrumented framework callback — `tracely.trace()` is a context marker rather than an active span, and LangGraph runs node functions with no span in context. A bare `set_state()` inside a LangGraph node warns rather than dropping the write; pass `span=` explicitly there, or just rely on the automatic capture.

---

## 2. Hermetic replay (`call_tool` / `call_llm` / `fixtures`)

These let the **same agent code** run live in production and deterministically offline in CI. Wrap each external call:

```python
def run(user_input: str):
    with tracely.agent("support-agent"):
        # In prod: calls the real fn and records the output. In replay: serves the recorded output.
        order = tracely.call_tool("get_order", lambda: get_order(order_id), args={"order_id": order_id})
        answer = tracely.call_llm("gpt-4o", lambda: chat(messages), input=messages, usage=(812, 96))
        return answer
```

- **Production (`--live` or no fixtures active):** `call_tool`/`call_llm` invoke your `fn`, record the output (and any error) on the span, and return it.
- **CI replay:** `tracely replay` activates the case's recorded **fixture bundle** via `with tracely.fixtures(bundle): ...`; `call_tool`/`call_llm` then **serve the recorded outputs in order** (or by `args` match) and never call your `fn`. A call that errored in production is reproduced on the span **and raised as `tracely.ToolError`**, so the agent's own `try/except` runs exactly as it would live — and the gate sees the same failure condition.

**Auto-instrument / drop-in code replays too** (no manual seam required): inside a `fixtures()` block Tracely class-patches the provider's create-method, so code that calls the SDK directly — `client.chat.completions.create(...)` under `instrument="auto"` or the `tracely_sdk.openai` drop-in — is served the recorded completion (reconstructed into a provider-shaped response) and never hits the network. Covered today: **OpenAI `chat.completions`** and **Anthropic `messages`**; other providers fall back to live in replay until added. The manual `call_llm` seam remains the way to get provider-agnostic hermetic replay everywhere.

This is what makes replay deterministic, offline, and free (no API keys, no cost). See [regression-testing design](https://github.com/Jwuthri/Tracely/blob/master/design/part2-tracely/05-regression-testing.md).

---

## 3. The CI gate CLI

```bash
tracely simulate [<agent> | --agent a,b | --all] [--min-pass-rate 0.9] [--timeout 900] [--github]
tracely gate     <agent> [--env ci] [--api …] [--key …] [--pr N] [--sha …] [--github]
tracely replay   <agent> (--entrypoint module:func | --cmd "…") [--live] [--github]
```

- **`tracely simulate`** — drive each agent's enabled **scenarios** (multi-turn conversations, or adversarial goals an attacker model improvises against) at the HTTP endpoint you registered for it, then gate on the result. **No agent code in CI** — Tracely makes the calls, so a TypeScript or Go service gates the same way a Python one does.
- **`tracely gate <agent>`** — gate a PR against **pre-emitted** `env=ci` traces (your CI already ran the agent and emitted traces); cases are matched to candidates by `input_digest`.
- **`tracely replay <agent>`** — re-run the agent **on each promoted case's recorded input** (fetched from `GET /api/gate/suite`), then gate. `--entrypoint module:func` calls a Python function per case; `--cmd "…"` runs a shell command per case (gets `TRACELY_INPUT`) that emits its own trace. Hermetic by default; `--live` makes real tool/LLM calls.

Scenarios belong to an agent, so `simulate` gates one agent, a comma-separated subset, or — with `--all` — every agent that has at least one **enabled** scenario (an agent whose suite is switched off is skipped, not reported untested). Each agent gets its own gate run but they share one commit status and one PR comment, and the job fails if any agent does. `--timeout` budgets the whole command.

All three **exit 0 (PASS) / 1 (FAIL) / 2 (no answer — timeout or unreachable API)** and, inside GitHub Actions (or with `--github`), post a **commit status + PR comment** with per-case results and soft warnings (latency/token deltas vs the last green gate). `--dry-run` prints the GitHub calls instead of sending; `--no-github` never touches GitHub. Config via flags or env (`TRACELY_API`, `TRACELY_KEY`, `TRACELY_AGENT`, `TRACELY_GATE_ENV`, `TRACELY_WEB_URL`, `GITHUB_TOKEN`). A reusable composite action lives at `.github/actions/tracely-gate/`.

---

## Examples (`sdk/examples/` + `sdk/example.py`)

[`examples/README.md`](examples/README.md) is the full index — **one runnable file per way of
tracing**, all the same fake-DB **two-agent conversation** (a Support Agent that hands the pricing
turn off to a Billing Agent, declaring its catalog via `tracely.trace(agents=...)`): each frontier
provider (OpenAI, Anthropic, Gemini, Mistral, Bedrock) + OpenRouter, each harness (LangChain
`create_agent`, LangGraph, LiteLLM, LlamaIndex, CrewAI), each first-party agent SDK (OpenAI Agents,
Claude Agent SDK, Google ADK), and each approach (`@observe`+`trace`, the `wrap_openai`/`wrap_anthropic`
drop-ins, manual spans). Highlights:

| File | Shows |
|---|---|
| `../example.py` | the minimal demo trace (agent → llm → failing tool). `make sdk-example`. |
| `examples/auto_openai.py` · `auto_anthropic.py` · `auto_gemini.py` · … | **automatic** provider tracing — zero span code (one file per frontier provider + OpenRouter). |
| `examples/auto_langchain.py` (`create_agent`) · `auto_langgraph.py` · `auto_litellm.py` · … | **automatic** harness tracing (one file per framework, current APIs). |
| `examples/auto_openai_agents.py` · `auto_claude_agent.py` · `auto_google_adk.py` | **automatic** first-party agent-SDK tracing (OpenAI Agents / Claude Agent SDK / Google ADK). |
| `examples/auto_agent.py` | **automatic** `@observe` + `trace()` agent → thinking/gen/tool tree. `make auto-agent`. |
| `examples/dropin_openai.py` · `dropin_anthropic.py` | non-patching `wrap_openai` / `wrap_anthropic` drop-ins. |
| `examples/manual_spans.py` | the manual escape-hatch API as a full agent (no provider/key needed). |
| `examples/weather_agent.py` / `weather_agent_cli.py` | a real agent wired with `call_tool`/`call_llm` for `tracely replay --entrypoint` / `--cmd`. |
| `examples/seed_conversations.py` | rich demo data using **every** SDK helper — single/multi-turn, multi-agent + handoffs, RAG (guardrail→embed→retrieve→chain), thinking, multimodal, structured output, multi-model. `make seed-demo`. |
| `examples/seed_multiturn.py` | one multi-turn conversation (manual API, no key) — the showcase for the **rolling summary** + **declared agents** (`@LIST_AGENT`). |
| `examples/seed_regression.py` | promote a failing trace → run red→green CI gates (fills Cases + Gates). `make seed-regression`. |
| `examples/seed_multicall.py` / `seed_handler.py` | repeated-call + handler examples for fixture replay. |

---

## Key decisions (and why)

1. **Thin wrapper, not a framework.** It only sets attributes on OTel spans — anyone already on OpenTelemetry/OpenInference can skip it and just emit the `tracely.*` hints. No lock-in.
2. **First-class `tracely.*` hints.** Agent/conversation/turn/step/env are emitted as semantic attributes so they become indexed columns server-side — the basis for agent-level evals and PR gating.
3. **`THINKING` is a span type, not a field.** Reasoning is its own observation, so it renders distinctly and carries its own token usage.
4. **Record-replay in the SDK.** `call_tool`/`call_llm` are the seam that makes "the recorded run is the test" real: the same code path records in prod and replays in CI, reproducing recorded errors via `ToolError` so error-handling behaviour is gated faithfully.
5. **The CLI is the CI contract.** `gate`/`replay` exit 0/1 and speak GitHub — so wiring Tracely into a pipeline is one step, with the hard gate being fail-to-pass and everything else advisory.
