Metadata-Version: 2.4
Name: botgauge-tracing
Version: 0.1.2
Summary: The BotGauge tracing SDK: trace AI agents in production, safely and privately. Built on OpenTelemetry.
Author-email: "BotGauge AI Inc." <support@botgauge.com>
License-Expression: FSL-1.1-ALv2
Project-URL: Homepage, https://www.botgauge.com/
Keywords: opentelemetry,otel,tracing,observability,llm,llm-observability,ai-agents,agents,langchain,openai,crewai,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Framework :: OpenTelemetry
Classifier: Framework :: OpenTelemetry :: Distros
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: botgauge-cli==0.1.2
Requires-Dist: opentelemetry-api>=1.26
Requires-Dist: opentelemetry-sdk>=1.26
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.26
Provides-Extra: openai
Requires-Dist: openinference-instrumentation-openai==0.1.65; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: openinference-instrumentation-anthropic==3.0.3; extra == "anthropic"
Provides-Extra: langchain
Requires-Dist: openinference-instrumentation-langchain==0.1.79; extra == "langchain"
Provides-Extra: langgraph
Requires-Dist: openinference-instrumentation-langchain==0.1.79; extra == "langgraph"
Provides-Extra: crewai
Requires-Dist: openinference-instrumentation-crewai==1.1.21; extra == "crewai"
Requires-Dist: openinference-instrumentation-openai==0.1.65; extra == "crewai"
Provides-Extra: llamaindex
Requires-Dist: openinference-instrumentation-llama-index==4.6.0; extra == "llamaindex"
Provides-Extra: google-adk
Requires-Dist: openinference-instrumentation-google-adk==1.0.4; extra == "google-adk"
Provides-Extra: openai-agents
Requires-Dist: openinference-instrumentation-openai-agents==2.5.3; extra == "openai-agents"
Provides-Extra: autogen-agentchat
Requires-Dist: openinference-instrumentation-autogen-agentchat==0.1.22; extra == "autogen-agentchat"
Provides-Extra: pydantic-ai
Requires-Dist: openinference-instrumentation-pydantic-ai==0.1.30; extra == "pydantic-ai"
Provides-Extra: mcp
Requires-Dist: openinference-instrumentation-mcp==2.0.13; extra == "mcp"
Provides-Extra: distributed
Requires-Dist: opentelemetry-instrumentation-requests>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-httpx>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-urllib3>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-asgi>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-wsgi>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-starlette>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-flask>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-django>=0.40b0; extra == "distributed"
Requires-Dist: opentelemetry-instrumentation-grpc>=0.40b0; extra == "distributed"
Provides-Extra: frameworks
Requires-Dist: botgauge-tracing[anthropic,autogen-agentchat,crewai,distributed,google-adk,langchain,llamaindex,mcp,openai,openai-agents,pydantic-ai]; extra == "frameworks"
Dynamic: license-file

<p align="center">
  <a href="https://www.botgauge.com/"><img src="https://www.botgauge.com/og_images/web-app-manifest-512x512.png" alt="BotGauge" width="80"></a>
</p>

<h1 align="center">botgauge-tracing</h1>

<p align="center">
  The BotGauge tracing SDK: trace AI agents in production, safely and privately. Built on OpenTelemetry.
</p>

<p align="center">
  <a href="https://pypi.org/project/botgauge-tracing/"><img src="https://img.shields.io/pypi/v/botgauge-tracing?style=flat-square&label=pypi%20botgauge-tracing" alt="PyPI version"></a>
  <a href="https://pypi.org/project/botgauge-tracing/"><img src="https://img.shields.io/pypi/pyversions/botgauge-tracing?style=flat-square" alt="Python versions"></a>
  <img src="https://img.shields.io/badge/OpenTelemetry-native-425CC7?style=flat-square" alt="OpenTelemetry native">
  <img src="https://img.shields.io/badge/license-FSL--1.1--ALv2-blue?style=flat-square" alt="License: FSL-1.1-ALv2">
</p>

`botgauge-tracing` is the BotGauge tracing SDK, built on the standard OpenTelemetry SDK and OTLP
exporter. Install it through the [`botgauge`](https://pypi.org/project/botgauge/) package and import
it as `from botgauge import tracing`. Its commands (`instrument`, `doctor`, `probe`, `setup`) run
through the `botgauge` command from [`botgauge-cli`](https://pypi.org/project/botgauge-cli/).

> **Early release (0.x).** Public API shapes may still change before 1.0.

## Install

```bash
pip install botgauge          # this SDK (Python library) and the `botgauge` command
botgauge setup --install      # adds tracing for the frameworks your app uses
```

`pip install botgauge` installs this SDK and the `botgauge` command. On its own it records the turns,
agents and spans your code creates. `botgauge setup --install` finds the frameworks in your
environment and installs the BotGauge extra for each (such as `botgauge[langchain]`), which records
their LLM and tool calls; `--detect` lists them without installing. You can also name the extra
yourself. The [`botgauge`](https://pypi.org/project/botgauge/) page lists every framework, and what to
do if yours is already instrumented or not listed.

## Get an API key

Sign in at [cloud.botgauge.com](https://cloud.botgauge.com) and create an API key (it starts with
`bg_`), then set it for your app: `export BOTGAUGE_API_KEY="bg_..."`. Questions:
[support@botgauge.com](mailto:support@botgauge.com).

## Quickstart

**In code:** record your own work with `turn()`, `agent()` and spans. To also record a framework's LLM
and tool calls, attach its instrumentor after `init()`, for example
`LangChainInstrumentor().instrument(tracer_provider=provider)` with `provider = tracing.init()`.

```python
from botgauge import tracing

tracing.init(api_key="...")  # or set BOTGAUGE_API_KEY and call tracing.init()

with tracing.turn(session_id="ses_123"):
    with tracing.agent("billing-agent"):
        tracer = tracing.get_tracer()
        with tracer.start_as_current_span("check_balance"):
            ...

tracing.flush()
tracing.shutdown()
```

**Zero-code**

```bash
pip install "botgauge[langchain]"  # install only the producer you need
export BOTGAUGE_API_KEY="..."
# export BOTGAUGE_ENDPOINT="https://your-collector/v1/traces"  # only for another collector
botgauge instrument python your_app.py
```

`botg` is a short alias for `botgauge` (`botg instrument python your_app.py`); `botgauge help` lists every
command the installed BotGauge packages provide.

## What you get

- One real OTel engine — no custom span class, no custom ID minting, no custom OTLP encoder.
- Never blocks the calling thread: spans export through a `BatchSpanProcessor`, off-thread.
- `tracing.agent()`/`tracing.turn()` stamp the frozen, dotted attribute names (see
  `src/botgauge/tracing/_attributes.py`) onto every span created inside them.
- Attaches to an existing `TracerProvider` (bring-your-own-OTel) rather than replacing it, and
  never creates a second one if one is already active.
- `flush()`/`shutdown()` enforce a hard deadline — a stuck exporter can't hang your process.
- A FULL/SKELETON/DROP topology filter keeps generic infrastructure spans (HTTP transport,
  database calls) out of your trace by default, bridging only what's structurally necessary to
  connect Agent→Agent hops (`BOTGAUGE_TRACE_SCOPE=all` to see everything).
- Privacy controls: `capture="metadata"|"text"|"full"` and `redacted=True|False`. See [Privacy](#privacy).
- With a customer `ConcurrentMultiSpanProcessor`, BotGauge runs its own start callbacks
  through a caller-context bridge, preserving Agent/Revision/Session/Turn identity and the
  customer's existing processor chain. The bridge is restored on shutdown; `doctor` reports
  whether it is active.
- Zero BotGauge baggage, ever. Standard W3C trace-context propagation only — your own propagator
  and baggage are never touched or replaced.

## Privacy

| Setting | Default | Options |
|---|---|---|
| `capture=` / `BOTGAUGE_CAPTURE` | `text`: prompts, responses and tool data, without images, audio or embedding vectors | `metadata` (structure, models, timing, token counts), `text`, `full` (adds media) |
| `redacted=` / `BOTGAUGE_REDACTED` | `True`: detected PII and credentials are masked before export | `False` exports original content in every capture mode, without masking |

The configured API key is always removed from exported data. Masking catches common formats, not
every possible sensitive value.

<details>
<summary><strong>Privacy details: what each mode selects, masking limits, zero-code settings</strong></summary>

Defaults are `capture="text", redacted=True`: prompts, responses and tool data, but no images,
audio or embedding vectors (most tracing platforms keep media bytes out of spans by default;
`capture="full"` exports them inline when you ask for it). `metadata` selects structural attributes
(IDs, models, timing, usage counts, HTTP/RPC/DB structure); `text` adds readable prompts,
responses, tools, errors, events, and link attributes while excluding recognized media and
embedding vectors; `full` captures all supported content. With `redacted=True`, background
cleaning masks detected PII and credentials only. Harmless SQL, URL parameters/fragments,
exception details, custom metadata, names, and scope versions remain readable when selected.
Sensitive values within SQL, DB URLs, URLs, events, and links are cleaned without hiding the
complete record. The configured key (`api_key=`, or a Bearer token passed in `headers=`) is
removed from protected export copies.
To select extra metadata keys, use `capture_allow_keys=["app.tenant_tier", "myco.*"]` or
`BOTGAUGE_CAPTURE_ALLOW_KEYS=app.tenant_tier,myco.*`; protected values are still cleaned.
**`redacted=False` overrides capture selection in every mode**: metadata/false, text/false,
and full/false export the same original supported FULL records, including sensitive data,
without privacy scanning or masking; like protected records, they are fitted to the sizes ingest
accepts (see [Long values and large spans](#long-values-and-large-spans)). FULL/SKELETON/DROP topology
filtering still applies; skeletons remain minimal. Raw export cannot recover content already
discarded by a customer-owned producer or exceed OTel's supported data types and limits.
BotGauge-owned producers use capture controls matching the resolved policy; raw configuration
disables BotGauge's own content hiding. Your `OPENINFERENCE_HIDE_*` settings still apply to them
in every mode and can only hide more. Customer-owned instrumentation and exporters remain intact.
Privacy patterns do not detect every sensitive fact or credential format. Protected values
retain documented technical size/work limits. Supported numeric PII is inspected across
integers, integral floats, numeric strings, and serialized structures. Recognized numeric
usage, cost, and duration measurements retain their values and types; arbitrary numeric IDs
still receive PII inspection. A subtree beyond the sanitizer's depth limit is replaced by
`[omitted: inspection limit]`, indicating a technical omission rather than detected PII.
Native containers have an 8-level limit and parsed structures a 64-level limit; these do not
apply with `redacted=False`. A list attribute keeps up to 2,000 elements in every mode, with
`redacted=False` too (ingest accepts 2,048 per span): its first quarter and its newest, with a
marker counting the rest when they are strings. Its elements share the attribute's size limit. A
map keeps as many entries as a span may send attributes, plus an `[omitted: inspection limit]`
entry counting the rest. Metadata omissions and text media
exclusions are capture selection, separate from sensitive-value masking.
Zero-code settings are `BOTGAUGE_CAPTURE=metadata|text|full` and
`BOTGAUGE_REDACTED=true|false`. Invalid capture values, including the previous `redacted`
spelling, fall back to protected metadata. Migrate to `capture="text", redacted=True`.
Invalid boolean environment values enable protection. Reconfigure export policy through
`shutdown()` and a new `init()`; restart zero-code processes to change producer capture
controls. Forks preserve both settings. `doctor` reports requested/effective capture
and protection without payloads or credentials.

Run each capture mode separately with protection enabled. Use the virtual environment containing
the updated SDK and your application's dependencies, with the endpoint/API-key settings above:

```bash
BOTGAUGE_CAPTURE=metadata BOTGAUGE_REDACTED=true botgauge instrument python your_app.py

BOTGAUGE_CAPTURE=text BOTGAUGE_REDACTED=true botgauge instrument python your_app.py

BOTGAUGE_CAPTURE=full BOTGAUGE_REDACTED=true botgauge instrument python your_app.py
```

Replace `your_app.py` with your script. Each command starts a new process. Metadata selects
structural content, text adds readable text while excluding recognized media/vectors, and full
selects all supported content. With `BOTGAUGE_REDACTED=true`, selected content receives sensitive
value cleaning. Setting `BOTGAUGE_REDACTED=false` enables original supported FULL content in all
three modes, bypassing their capture exclusions and privacy cleaning.

</details>

### Long values and large spans

BotGauge's ingest accepts attribute values up to 1 MiB and up to 1,040 attributes on a span (its
`BOTGAUGE_MAX_ATTRIBUTE_VALUE_BYTES` and `BOTGAUGE_MAX_ATTRIBUTES_PER_ENTITY`), and 128 events and
128 links. A conversation history of up to about 250,000 tokens is exported whole. BotGauge fits
what is larger rather than have ingest drop it, in every capture mode and with `redacted=False`
too:

- A value over the limit keeps its start and its end, with `[omitted: N characters]` between them.
  JSON keeps its structure and its keys. A long conversation keeps, in this order: the current
  question and everything after it (the tool calls and results that answer it), its opening (the
  system instructions and the first request), then its newest history. Only messages in the middle
  are left out, as few as the limit allows, and `[omitted: N items]` counts them. A message too
  large for its share keeps its start and its end; a short question is never shortened for a long
  tool result. With `redacted=True`, every part kept is cleaned with the text around it, so a
  secret on a cut is never exported in part.
- A span with more attributes than ingest accepts keeps every attribute that isn't part of a
  flattened list such as `llm.input_messages.N.*`, and of each list its first element and its
  newest ones. A span with more than 128 events keeps its first and its newest. What is left out is
  counted in the span's dropped counts.
- `botgauge.span.fitted_attributes` names the attributes fitted on a span (a flattened list by its
  prefix, such as `llm.input_messages`), so a reader can tell a complete capture from a fitted one.
- A span's content is limited to 4 values' worth (4 MiB), its largest values giving way first, and
  a large batch is sent in several requests.

Set `BOTGAUGE_MAX_ATTRIBUTE_VALUE_BYTES` and `BOTGAUGE_MAX_ATTRIBUTES_PER_ENTITY` to the same numbers
as the ingest you send to, if it differs, for example `65536` and `256` for an ingest that has not
raised them yet. They are read when `tracing.init()` runs, so no new SDK version is needed. Values
are fitted 4 KiB below the value limit, and 16 fewer attributes are sent than the attribute limit,
leaving room for what ingest adds. `botgauge doctor` shows the limits in effect.

Two attribute limits apply, one after the other. While your app records a span, a provider
BotGauge creates holds up to 1,024 attributes, or as many as export may send if that is more
(unless `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` or `OTEL_ATTRIBUTE_COUNT_LIMIT` is set), so nothing is lost
before export chooses. OpenTelemetry's default is 128 while recording, after which it drops the
oldest attributes, such as the span kind and the first messages of a long conversation. If you
pass your own provider, give it `span_limits=SpanLimits(max_span_attributes=1024)` for long agent
conversations.

## Configuration

### BotGauge environment variables

Arguments to `tracing.init()` win over these. "Both" means `tracing.init()` and
`botgauge instrument`; "instrument" means only the zero-code launcher reads it (with `init()`, use
the matching argument).

| Variable | Read by | Default | Effect |
|---|---|---|---|
| `BOTGAUGE_API_KEY` | Both | unset | Your BotGauge key, sent only as `Authorization: Bearer <key>`. The one setting you need |
| `BOTGAUGE_ENDPOINT` | Both | `https://api.botgauge.com/v1/traces` when a key is set | Full OTLP/HTTP traces URL of the collector to send to |
| `BOTGAUGE_CAPTURE` | instrument | `text` | `metadata`, `text` or `full`; see the privacy notes above |
| `BOTGAUGE_REDACTED` | instrument | `true` | `false` exports original content without masking, in every capture mode: it overrides `BOTGAUGE_CAPTURE` |
| `BOTGAUGE_CAPTURE_ALLOW_KEYS` | Both | empty | Comma-separated extra attribute keys to keep, globs allowed (`app.tenant_tier,myco.*`); merged with `capture_allow_keys=` |
| `BOTGAUGE_TRACE_SCOPE` | Both | filtered | `all` exports infrastructure spans (HTTP, database) in full instead of filtering them |
| `BOTGAUGE_STRICT` | instrument | off | `1` stops the app from starting on instrumentation problems (same as `--strict`) |
| `BOTGAUGE_VERIFIED_ONLY` | instrument | off | `1` traces only framework versions BotGauge verified |
| `BOTGAUGE_PRODUCER_OWNER` | instrument | empty | `langchain=customer` (comma-separated) leaves a framework you already instrument to you |
| `BOTGAUGE_ALLOW_INSECURE_HTTP` | Both | `false` | `true` allows sending the key to a remote `http://` endpoint. Local addresses need no opt-in |
| `BOTGAUGE_CONSOLE_EXPORT` | Both | off | `1` prints spans to stdout when no key or endpoint is set. Local development only |
| `BOTGAUGE_MAX_ATTRIBUTE_VALUE_BYTES` | Both | `1048576` (1 MiB) | The largest attribute value your BotGauge ingest accepts, in bytes or with a `KiB`/`MiB` unit, from 8 KiB to 4 MiB. Set it to the same number as ingest; values are fitted 4 KiB below it (see [Long values and large spans](#long-values-and-large-spans)) |
| `BOTGAUGE_MAX_ATTRIBUTES_PER_ENTITY` | Both | `1040` | The most attributes your BotGauge ingest accepts on a span, event, link, resource or scope, from 64 to 16,384. Set it to the same number as ingest; 16 fewer are sent |


<details>
<summary><strong>OpenTelemetry environment variables that affect BotGauge</strong></summary>

BotGauge is built on the OpenTelemetry SDK, so some standard variables affect it too. Where a row
says "when BotGauge creates the provider", it applies when you call `tracing.init()` without
`tracer_provider=` and no provider is set up yet; otherwise your own provider's configuration
applies.

| Variable | Effect on BotGauge |
|---|---|
| `OTEL_SDK_DISABLED=true` | **Turns tracing off.** No spans are recorded or exported, with or without your own provider. Use it to disable BotGauge in tests or one environment without changing code |
| `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES` | Added to the exported resource when BotGauge creates the provider |
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Apply when BotGauge creates the provider and `sample_ratio=` is not given. A sampler set for another tool therefore samples BotGauge's traces too; BotGauge prints a one-time warning at startup when it will drop traces, and `botgauge doctor` shows the active sampler. Pass `sample_ratio=1.0` to record everything regardless |
| `OTEL_BSP_SCHEDULE_DELAY`, `OTEL_BSP_MAX_QUEUE_SIZE`, `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BSP_EXPORT_TIMEOUT` | Tune the batching of BotGauge's own export. An unusable value is logged once on the `botgauge` logger and OpenTelemetry's default is used |
| `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` and the other span limit variables | Apply when BotGauge creates the provider. Without them, a span keeps up to 1,024 attributes, and BotGauge fits exported values itself (see [Long values and large spans](#long-values-and-large-spans)). A value length limit keeps only the first characters of each value, before BotGauge sees it, so a conversation history loses its newest turns: leave it unset; BotGauge fits values to `BOTGAUGE_MAX_ATTRIBUTE_VALUE_BYTES` |
| `OTEL_PROPAGATORS` | Which headers carry trace context between services. BotGauge never replaces your propagator. For a service with untrusted callers, see [Public edges](#public-edges). The values for this and the sampler variables are in [OpenTelemetry's SDK configuration](https://opentelemetry.io/docs/languages/sdk-configuration/general/) |
| `OTEL_EXPORTER_OTLP_*` | **Ignored** by BotGauge's exporter. They stay with your own exporters, which never receive the BotGauge key |

</details>

### Logs

BotGauge reports problems (a rejected key, an invalid setting, an untraced framework) as warnings on
the standard Python logger named `botgauge`. Route or silence them like any other logger, for
example `logging.getLogger("botgauge").setLevel(logging.ERROR)`. For a full picture of what is
active, run `botgauge doctor`, or call `tracing.doctor.report()` inside your app.

## Reference

<details>
<summary><strong>Credentials, endpoints and local development</strong></summary>

`init()` sends the key as `Authorization: Bearer <key>`, BotGauge's only credential. An
`Authorization` header passed in `headers=` replaces it. Arguments win over the environment:
without `api_key=` or `endpoint=`, `init()` reads `BOTGAUGE_API_KEY` / `BOTGAUGE_ENDPOINT`, and a key with no
endpoint exports to `https://api.botgauge.com/v1/traces`. It never reads `.env` files (load them
in your app, e.g. with python-dotenv) or `OTEL_EXPORTER_OTLP_*`. The first 401/403 from the
endpoint logs one `botgauge: BotGauge rejected the API key (...)` line with the server's reason.
The line names the Authorization header instead when `headers=` supplied it, and says that no
credential was sent when an endpoint is set without a key.

BotGauge disables export when a key or custom headers would be sent to a remote `http://`
endpoint, or to an endpoint with no `http://` or `https://` scheme. Local HTTP collectors remain
available. Use `https://` for remote collectors, and add a scheme to an endpoint that has none. If
you deliberately need remote cleartext HTTP, set `BOTGAUGE_ALLOW_INSECURE_HTTP=true`; BotGauge
still warns that the connection sends headers without transport encryption. Ambiguous URL
authorities are treated as remote.

With neither a key nor an endpoint, BotGauge emits a configuration notice to stderr and
discards its exports; it does not print spans. For local development only, set
`BOTGAUGE_CONSOLE_EXPORT=1` to explicitly send sanitized spans to stdout. A customer-owned
OTel exporter attached to the same provider continues to operate independently.

</details>

<details>
<summary><strong>Streaming generators</strong></summary>

**Streaming generators:** Keep a `turn()` scope within one execution context. Do not
leave it open across `yield` when a framework may resume each chunk in a fresh
context (for example, a synchronous generator consumed through Starlette's thread
pool). Later chunks can lose their turn and parent span, and closing the scope can
log `Failed to detach context`. Finish traced work before yielding, or keep the
entire iteration in one context. If chunks run in separate contexts, start and end
each tracing scope within the context that processes that chunk; preserving one Turn
across the whole stream requires the application to preserve its context as well.

</details>

<details>
<summary><strong>Zero-code: <code>.env</code> files and the export destination</strong></summary>

Or keep the settings in a `.env` file: `botgauge instrument --env-file .env python your_app.py`.
It reads `KEY=VALUE` lines the way `uv run --env-file` and `docker run --env-file` do: comments,
an optional `export ` prefix, and quotes are handled, and variables aren't expanded. It never
overrides a variable that is already set, so real deployment settings win. The flag can be
repeated; an earlier file wins over a later one. Manual `tracing.init()` does not read `.env`;
load it in your app (e.g. with python-dotenv).

Zero-code export reads only `BOTGAUGE_ENDPOINT` (default `https://api.botgauge.com/v1/traces`
when `BOTGAUGE_API_KEY` is set) for its destination. Customer
`OTEL_EXPORTER_OTLP_*` settings are left to the customer's OTel exporters in both
directions: BotGauge never uses them as a fallback destination for its connection token, and
BotGauge's own exporter ignores them, so customer collector headers, TLS files, timeout, and
compression are never applied to BotGauge export (which always uses gzip). With neither
variable set, BotGauge export is disabled by default and a configuration notice is
written to stderr; the explicit development-only `BOTGAUGE_CONSOLE_EXPORT=1` setting
still enables console output.

</details>

<details>
<summary><strong>Framework selection, verified versions and strict mode</strong></summary>

Selects certified framework producers from the versioned compatibility registry and activates
compatible OTel transport instrumentors via standard `opentelemetry_instrumentor` entry points.
The registry covers pinned LangChain/LangGraph, direct OpenAI, direct Anthropic,
LlamaIndex+OpenAI, CrewAI, OpenAI Agents, Google ADK, AutoGen AgentChat and MCP (each with only its own producer).
Certification records the versions BotGauge verified, as (framework, instrumentor) pairs: a
framework version is verified only with an instrumentor version it passed certification with. Only
those exact pairs are verified, never a range: a version older than the oldest verified, between
two verified versions, newer than the latest verified, or a pre-release is not verified. Verified
pairs stay verified when newer ones are added. It is not a requirement. An installed framework at
a version that isn't verified is still traced, and startup logs one line per framework saying why:
`botgauge: tracing openai at a version BotGauge has not verified: openai 3.40.0 is newer than the
latest version BotGauge verified (3.35.1); verified: ...`. Inside the app,
`tracing.doctor.report()["instrumentors"]["not_verified"]` returns them; a separate `botgauge doctor`
process cannot see the app's activation. Set `BOTGAUGE_VERIFIED_ONLY=1` to trace only verified
versions. Versions the registry lists as known to break tracing are never traced, and startup says
why. AI instrumentors that aren't in the registry are still skipped.
BotGauge checks
the public OTel already-instrumented marker and supports explicit customer ownership via
`BOTGAUGE_PRODUCER_OWNER=langchain=customer`. Set `BOTGAUGE_STRICT=1` to make selection failures
and versions that aren't verified fatal for CI (or use `botgauge instrument --strict python your_app.py`):
the launcher logs `botgauge: --strict: not starting the application: <reasons>` and exits 1
before the app runs. Ownership also accepts registered entry-point names such as
`llama_index=customer` (the canonical target is `llamaindex`); unknown names produce a warning.
A producer that traces several frameworks (the LangChain instrumentor also traces LangGraph) stays
off when the customer owns any of them; the others are then not traced by BotGauge, so they are not
reported as not verified, and importing one logs missing-producer guidance that names that ownership.
An installed producer extra without its framework is reported as `absent` and does not fail
strict mode. Resolve actual selection findings or disable strict mode; setting
`BOTGAUGE_VERIFIED_ONLY=1` alone does not clear strict failures. Strict mode also stops the app
if the target Python cannot import BotGauge
or if interpreter options `-I`, `-E`, or `-S` would bypass the startup hook. When an app imports a
registered framework without an active producer, startup logs one recommendation for the
BotGauge extra, or points to `botgauge doctor` if the extra is installed but did not activate. Client
libraries that frameworks import themselves (`openai`, `anthropic`, `mcp`) get no hint while a
framework producer is active, so `--strict` does not fail a correctly traced framework app;
importing `langchain_core` (as `langchain_openai` does) counts as importing LangChain. Unknown
frameworks are not guessed.
If a missing producer is discovered on a later framework import, strict mode logs the
guidance and exits the whole process with code 1, including when the import occurs in a
worker thread; the application may already have started by then.
Selected instrumentors use the same FULL/SKELETON/DROP filtering and privacy
pipeline. Works with `python -m pkg`, any `python3.x` binary, sibling imports next to your script,
and child processes your app spawns (they inherit instrumentation the same way `gunicorn --preload`
workers would). The directory you run `botgauge instrument` from is importable by the launched app,
after the standard library and installed packages so it can't shadow them. Unlike
`opentelemetry-instrument`, it is not put on `PYTHONPATH`: child processes don't inherit it, a
`sitecustomize.py` in it is never run, and `python -P` or `PYTHONSAFEPATH` keep it off the path.

</details>

<details>
<summary><strong>Implicit turns in zero-code mode</strong></summary>

Zero-code mode opens an implicit Turn only at a top-level certified **Agent** span, with no
active `tracing.turn()` and no inbound remote parent. An app that only runs
chains or LLM calls (a plain LangChain `prompt | llm`, or a LangGraph graph, whose root span is a
CHAIN) produces no Agent span, so its spans carry no `botgauge.turn.*` attributes; wrap each
interaction in `tracing.turn()` to group it. An Agent label set after a span starts is read when a
nested Agent decides whether it is top-level, but a label set on an outer span only after a nested
Agent's Turn decision (at its start, or at its end if it was labelled late) cannot move the root back
to the outer span; set the outer Agent's label at start, or use `tracing.turn()`.

</details>

<details>
<summary><strong>Forking servers (gunicorn --preload, multiprocessing)</strong></summary>

Forking servers (`gunicorn --preload`, multiprocessing): under `botgauge instrument`, each forked
worker rebuilds its own BotGauge pipeline automatically (new exporter connection and batch thread,
same settings), so the application needs no `post_fork` hook. With manual `tracing.init()`, call
`init()` again after the fork, as before. Either way, a fork-context `multiprocessing` or billiard worker sends
its queued spans when it exits normally, waiting at most one second (none when export is already
failing). A pool closed with `terminate()`, which includes leaving a `with Pool(...) as pool:` block,
stops its workers before that, so their spans are lost: use `pool.close()` and `pool.join()`, or
call `tracing.flush()` at the end of each task. On macOS only, a forked worker that makes any HTTP request
after the parent has started threads is killed by the Objective-C runtime; this affects preload apps
with or without BotGauge, and not Linux. For local development on macOS, set
`OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` in the server's environment.

</details>

<details>
<summary><strong>PydanticAI</strong></summary>

PydanticAI is a zero-code producer at its verified versions (PydanticAI 1.25.1 and 2.57.0 with
`openinference-instrumentation-pydantic-ai` 0.1.30, and 1.25.1 with 0.1.27). It instruments itself natively and
OpenInference ships a span processor rather than an instrumentor, so BotGauge activates it
through its own `botgauge.native_producers` entry point: `botgauge instrument` calls
`Agent.instrument_all()` on BotGauge's provider (GenAI span names, text content recorded with
`capture="text"` or `"full"`, and with every raw override) and attaches `OpenInferenceSpanProcessor`. If your application already
instruments PydanticAI, BotGauge reports a duplicate and leaves it unchanged; set
`BOTGAUGE_PRODUCER_OWNER=pydantic-ai=customer` to declare that ownership explicitly. The PydanticAI 1.25
cohort runs on OTel 1.39.1, which conflicts with the currently tested non-PydanticAI multi-extra
OTel 1.44.0 constraints. A resolver dry-run with only PydanticAI's OTel 1.39.1 constraints
succeeds for `[frameworks]` (every instrumentor at once), but no installed combined runtime,
`pip check`, or framework tests have been validated. Do not infer that `[frameworks]` or a mixed
PydanticAI producer is verified merely
because the extra resolves.

</details>

<details>
<summary><strong><code>doctor</code>, <code>setup</code> and <code>probe</code></strong></summary>

```bash
botgauge doctor   # read-only report: what's active, what's missing, no side effects
botgauge doctor --strict  # exit 1 for uninitialized tracing, propagation or selection problems
botgauge doctor --json    # the same read-only report as machine-readable JSON
```

`doctor` observes only the current process: running it separately from your app cannot inspect
the app's activation report. `doctor --strict` does not emit a probe span or mutate instrumentation.

`botgauge setup --detect` lists recommended producer extras without importing frameworks or installing anything. `botgauge setup --install` explicitly installs the listed extras with pip; no automatic installation occurs on startup.

**Explicit ingest diagnostic**

```bash
BOTGAUGE_API_KEY="..." botgauge probe --endpoint https://your-collector/v1/traces
```

Unlike `doctor`, **`probe` sends data**: one metadata-only `botgauge.probe` span through the
BotGauge privacy/filter/batch pipeline to the specified OTLP HTTP endpoint. It requires an
explicit URL (no console fallback), runs in its own process, and exits nonzero if ingest
acceptance cannot be confirmed. A success confirms OTLP export was acknowledged, **not** that a
downstream worker normalized or stored the span. The key is read from the environment, not CLI
arguments. Never point it at production without authorization; it creates a real trace.

</details>

<details>
<summary><strong>Sampling and attribute size limits</strong></summary>

`tracing.init(sample_ratio=0.5)` enables opt-in head sampling for a provider BotGauge creates; default is record everything, unless `OTEL_TRACES_SAMPLER` is set, in which case BotGauge says so at startup (see [Configuration](#configuration)). With the default sampler and with `sample_ratio` (both parent-based, as in OpenTelemetry), a request that continues a caller's trace follows the caller's sampling decision (the last two characters of its `traceparent`): `-00` is not recorded, and `-01` is recorded whatever `sample_ratio` says. An `OTEL_TRACES_SAMPLER` that isn't parent-based, such as `always_on` or `traceidratio`, decides on its own. For a service with untrusted callers, see [Public edges](#public-edges). After `shutdown()`, re-initialization may specify the same ratio for that BotGauge-created global provider. A different ratio or an existing customer provider is rejected because its sampler cannot be changed. A head-dropped trace can't be recovered downstream, and policy/authority/verifier evidence must never depend on sampling. Attribute values over the ingest limit (1 MiB by default) are fitted, keeping their start and their end (see [Long values and large spans](#long-values-and-large-spans)); TTFT/streaming fields are reserved in the attributes contract but not automatically measured by these isolated producers.

</details>

## Public edges

If your application accepts requests from an untrusted or public caller, that caller's own
`traceparent` header should not be trusted as the real parent span — an attacker could otherwise
inject an arbitrary trace/span ID with no authority consequences intended.

**If the service runs server or MCP instrumentation, turn off incoming trace context for the
whole process.** That includes `botgauge instrument` and any web framework instrumentation you add
(Flask, Django, FastAPI and other ASGI/WSGI servers, gRPC). These start the request's span from the
caller's `traceparent`, and the MCP instrumentation from a request's `_meta` field, over stdio as
well as HTTP, before your code runs. So the caller decides:

- the trace ID of everything the request records;
- whether the request is recorded at all: with OpenTelemetry's default sampler, a `traceparent`
  ending in `-00` turns recording off for the whole request, and `-01` turns it on whatever
  `sample_ratio` says.

`extract_context()` (below) can't undo this, because the caller's context is already current when
it runs. Turn incoming trace context off instead:

- `OTEL_PROPAGATORS=none`, on OpenTelemetry 1.33 or newer. Older versions don't accept `none` and
  stop at startup with "Propagator none not found".
- On any version, once at startup:
  ```python
  from opentelemetry.propagate import set_global_textmap
  from opentelemetry.propagators.composite import CompositePropagator

  set_global_textmap(CompositePropagator([]))
  ```

Both also stop the service sending trace context to the services it calls, so use them only in
processes that face untrusted callers. For HTTP only, removing `traceparent`, `tracestate` and
`baggage` from incoming requests in middleware that runs before the instrumentation also works.

**Without server or MCP instrumentation**, where your code reads the headers itself, extract
context at that specific boundary (e.g. one public-facing route, not your whole process) with
`trust_incoming_context=False`:

```python
from botgauge import tracing

ctx, link = tracing.extract_context(request.headers, trust_incoming_context=False)
tracer = tracing.get_tracer()
with tracer.start_as_current_span("handle_request", context=ctx, links=[link] if link else []):
    ...
```

With `trust_incoming_context=False`, `extract_context()` retains the current local context for parenting
and returns a link to a valid incoming span. The example attaches that link with
`botgauge.link.relation="continuation"`; the untrusted claim is not used as the parent. A new
trace starts only when no valid local parent is active. Internal/trusted service-to-service
hops should keep the default (`trust_incoming_context=True`), which is what most calls need.

## Known limits

- Each framework is verified on its own, at exact framework and instrumentor versions; startup names
  any installed version that isn't one of them. Combined or overlapping producer profiles are not verified.
- A direct OpenAI + Anthropic mixed-producer test covers shared turn context and
  duplicate/ownership handling. With LangChain and direct OpenAI both active, explicit ownership
  emits one LLM span; activating both producers emits two sibling LLM spans on the same client.
- PydanticAI 1.25.1 needs OpenTelemetry 1.39.1 in an isolated environment, so it is not part of
  the compatible multi-extra set.
- A nightly job tests the latest framework versions; a passing drift test never expands the
  pinned compatibility registry by itself.
