Metadata-Version: 2.4
Name: bir-sdk
Version: 0.2.0
Summary: Minimal local tracing SDK for LLM applications.
Author: mskayacioglu
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/bir-ai/bir-python
Project-URL: Documentation, https://github.com/bir-ai/bir-python/tree/main/docs
Project-URL: Source, https://github.com/bir-ai/bir-python
Project-URL: Issues, https://github.com/bir-ai/bir-python/issues
Keywords: ai,evals,llm,observability,tracing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5; extra == "docs"
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.20; extra == "otel"
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20; extra == "otel"
Dynamic: license-file

# Bir Python SDK

Minimal, zero-runtime-dependency, local-first tracing and evals for Python LLM
applications.

Bir records traces, spans, generations, tool calls, retrievals, and scores to
local JSONL without requiring a server. Start locally, evaluate deterministic
regressions, and send events to a Bir server when you want to inspect them in a
dashboard.

## Installation

```bash
python -m pip install bir-sdk
```

The distribution name is `bir-sdk`; the import name is `bir`. Runtime
installation has no third-party dependencies. Bir also ships inline type
annotations and a PEP 561 `py.typed` marker.

An opt-in `otel` extra (`pip install 'bir-sdk[otel]'`) adds an OpenTelemetry/OTLP
exporter that forwards recorded traces to an existing observability backend; the
runtime install stays dependency-free without it. See
[Forwarding traces to OpenTelemetry](#forwarding-traces-to-opentelemetry).

## Quickstart

```python
from bir import generation, observe, score


@observe()
def answer_question(question: str) -> str:
    with generation("local.llm", model="demo-model") as gen:
        response = f"Answer: {question}"
        gen.set_output(response)
        gen.set_usage(input_tokens=12, output_tokens=24)
    score("helpfulness", 0.82)
    return response
```

Events are written to `.bir/traces.jsonl` by default. Input and output capture
is disabled unless you explicitly enable it.

Inspect them from the command line without a server: `bir traces` lists recorded
traces, `bir show <trace-id>` prints one trace as an indented tree of its spans,
generations, tool calls, and scores, and `bir stats` summarizes trace counts,
token usage, cost per currency, and latency (count, mean, and p95) for a quick
cost or health check (add `--json` to any of them for a structured form). See
[CLI & environment](docs/site/cli-env.md).

When capture is on, Bir redacts common secret-like fields and text before
anything is written. Those built-in rules always apply and cannot be turned off,
but you can widen them for your own credential names and formats with
`configure(additional_secret_keys=[...], additional_redaction_patterns=[...])`.
See [capture and privacy](docs/site/capture-privacy.md).

## Correlating your logs with traces

Read the active ids with `get_current_trace_id()` and `get_current_span_id()` to
stamp your application's own logs and metrics, so they line up with Bir traces
later. Both return `None` outside any trace and never raise:

```python
import logging

from bir import get_current_span_id, get_current_trace_id, observe


@observe()
def answer(question: str) -> str:
    logging.info(
        "handling question",
        extra={"trace_id": get_current_trace_id(), "span_id": get_current_span_id()},
    )
    return "ok"
```

`get_current_trace_id()` returns the active trace root id and
`get_current_span_id()` the innermost open span, generation, or tool call (the
trace root when none is open). The values match the `trace_id`/`parent_id` later
written to the JSONL, and each asyncio task and thread sees its own ids. The
accessors are read-only — there is no setter and no context is exposed for
injection or cross-process propagation.

## Attaching metadata discovered mid-body

Every trace-work context manager — `trace()`, `span()`, `generation()`,
`tool_call()`, and `retrieval()` — exposes `set_metadata(...)` so you can record
context that only becomes known while the body runs (a resolved route, a
cache-hit flag, a request id) before the event is written:

```python
from bir import generation, observe


@observe()
def answer(question: str) -> str:
    with generation("local.llm", model="demo-model", metadata={"provider": "demo"}) as gen:
        response = f"Answer: {question}"
        gen.set_metadata({"route": "fast", "cache_hit": False})
        gen.set_output(response)
    return response
```

`set_metadata()` merges into any metadata passed at creation time — later keys
win, including across repeated calls — and the merged metadata is redacted before
it is written with the same rules as captured input and output, so secret-like
fields never reach the JSONL. It works with both `with` and `async with`, and the
argument must be a mapping.

## Estimating cost from a local price table

Cost is user-provided by default — Bir bundles no prices because provider prices
go stale. If you would rather not call `set_cost()` on every call, supply your own
per-token rates once with `configure(model_prices=...)` and Bir fills the cost
from token usage:

```python
from bir import configure, generation, observe

configure(
    model_prices={
        "gpt-4o-mini": {"input": 0.00000015, "output": 0.0000006},
        "mistral-large": {"input": 0.000002, "output": 0.000006, "currency": "EUR"},
    }
)


@observe()
def answer(question: str) -> str:
    with generation("chat", model="gpt-4o-mini") as gen:
        gen.set_usage(input_tokens=1000, output_tokens=400)
        # No set_cost(): input_cost/output_cost/total_cost are derived from the rates.
    return "ok"
```

Each entry sets a non-negative `input` and/or `output` per-token rate plus an
optional `currency` (default `USD`). The cost is derived only for a generation
that has the matching token counts and no explicit `set_cost()` — an explicit cost
always wins, and a usage without the needed token split is left without a cost.
The table is validated at `configure()` time, ships no bundled prices, and keeping
the rates current is your responsibility. With no table configured, cost behavior
is unchanged. See [core API](docs/site/core-api.md).

## Tracing generators and streaming

`@observe()` also traces generator and async-generator functions across their
full iteration, not just their creation. The wrapper stays lazy — the body does
not run and nothing is written until the first iteration — and the trace stays
open from the first `next()`/`await __anext__()` through exhaustion, so spans and
generations created in the body (for example while consuming a streamed LLM
response) attach to it:

```python
from bir import generation, observe


@observe()
def stream_answer(question: str):
    with generation("local.llm", model="demo-model") as gen:
        chunks = []
        for token in ("Ans", "wer", "!"):
            chunks.append(token)
            yield token
        gen.set_output("".join(chunks))
```

The trace is finalized when the generator is exhausted (a successful trace),
raises (a redacted error, re-raised unchanged to the consumer), or is closed or
cancelled early. An early `close()`/`aclose()` or a cancellation is recorded as a
successful trace whose `metadata.generator.outcome` is `"closed"`, and it resets
all trace context so nothing leaks into later work. `send`/`throw`/`close` (and
`asend`/`athrow`/`aclose`) and the body's `finally` blocks all behave exactly as
they would without the decorator, and concurrent async generators running in
separate tasks stay isolated. Yielded values are never buffered; with output
capture enabled only a bounded yielded-item count is recorded under
`metadata.generator.items`.

The optional [provider integrations](docs/site/integrations.md) ship async
counterparts (`trace_chat_completion_async`, `trace_messages_async`,
`trace_completion_async`, and so on) for async clients such as `AsyncOpenAI`,
`AsyncAnthropic`, and `litellm.acompletion`. Each awaits the provider coroutine
inside an active trace and records one generation; the streaming wrappers resolve
to an async iterator you consume with `async for`, never buffering the stream. The
synchronous wrappers likewise accept `stream=True` — yielding the provider's
chunks unchanged and recording the accumulated text and final token usage once
the stream is consumed — across OpenAI (Chat Completions and Responses),
Anthropic, Gemini, Mistral, Cohere, LiteLLM, and Vertex AI. AWS Bedrock's Converse
stream is a distinct method rather than a `stream=True` flag, so it has its own
`trace_converse_stream` wrapper that yields the stream's events unchanged and
records the same way.

For agent frameworks, Bir ships dependency-free callback handlers that map a
framework's own events into Bir traces without importing the framework:
`BirCallbackHandler` for LangChain, `BirLlamaIndexHandler` for LlamaIndex, and
`BirAgentsTracingProcessor` for the OpenAI Agents SDK. The Agents processor
implements the SDK's tracing-processor interface, turning an agent run into a Bir
trace whose model spans become generations and tool spans become tool calls;
register it with `agents.add_trace_processor(BirAgentsTracingProcessor())`.

## Local persistence and concurrency

Trace appends and size-based rotation are serialized across threads and local
processes that write the same trace path. Opt-in sent-ID bookkeeping uses a
separate lock around sidecar merge and replacement, so concurrent workers and
`bir send` processes preserve the union of accepted IDs. The implementation is
stdlib only: it uses `flock` on POSIX and byte-range locking on Windows, with
stable hidden lock files beside the trace and sidecar files.

These locks are advisory, so every writer must use Bir's persistence path.
Cross-host coordination and filesystems that do not implement normal local
advisory-lock semantics are not supported; use one local trace path per host in
those deployments. Lock files may remain on disk and must not be deleted while
Bir processes are active.

By default `send_events()` and `bir send` upload only the active trace file. Pass
`include_rotated=True` (or `bir send --include-rotated`) to also upload retained
size-rotated files oldest-first, deduplicated by event ID, so rotation does not
strand unsent events. Both `send_events()`/`bir send` and
`send_experiment()`/`bir send-experiment` retry transient failures (network
errors, timeouts, and HTTP 5xx) with bounded exponential backoff via `retries`
and `backoff`, while HTTP 4xx and malformed inputs fail immediately. See
[server uploads](docs/site/sending.md).

## Forwarding traces to OpenTelemetry

If you already run an OpenTelemetry backend, you can replay locally recorded Bir
traces as OpenTelemetry spans and ship them over OTLP. This is opt-in and never
runs on its own: nothing imports `opentelemetry` until you call the exporter, and
it only reads your local JSONL — it never writes to or alters it.

Install the extra, then forward loaded traces:

```bash
python -m pip install 'bir-sdk[otel]'
```

```python
from bir import load_traces
from bir.integrations.otel import export_traces_to_otlp

export_traces_to_otlp(
    load_traces(),
    endpoint="http://localhost:4318/v1/traces",
    service_name="rag-api",
)
```

`export_traces_to_otlp()` also accepts a single `LoadedTrace`, an iterable of
them, or a path to a trace file (loaded via `load_traces`). Each Bir trace
becomes one OpenTelemetry trace: the trace root maps to a root span and every
other event maps to a child span linked by `parent_id`, carrying over start/end
times and `success`/`error` status. Attributes follow the GenAI semantic
conventions where they exist (`gen_ai.request.model`,
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`) with `bir.*`
attributes for the rest (event type, score value, token totals, cost, and the
originating Bir ids). Pass `headers=` for backend auth, or inject your own
configured `span_exporter=` for a different transport. Calling the exporter
without the extra installed raises a clear error pointing you to
`pip install 'bir-sdk[otel]'`.

## Evaluations and experiments

Bir ships deterministic, local-first evaluators and an experiment runner that
scores a task over a dataset and persists per-example results and a summary under
`.bir/experiments/`. Use `run_experiment()` for synchronous tasks, or
`run_experiment_async()` when your task is a coroutine such as an async provider
client:

```python
import asyncio

from bir.evals import Dataset, DatasetExample, contains, run_experiment_async


async def answer(question: str) -> str:
    ...  # await your async model client


result = asyncio.run(
    run_experiment_async(
        "prompt-v1",
        dataset=Dataset([DatasetExample(id="q1", input={"question": "Hi"})]),
        task=answer,
        evaluators=[contains("Hi")],
        max_concurrency=8,
    )
)
```

`run_experiment_async()` runs up to `max_concurrency` examples concurrently while
keeping results, JSONL rows, and summary aggregates in dataset order. It accepts
async tasks, plain sync callables, and sync callables that return an awaitable,
and otherwise matches `run_experiment()`. See
[local evals and experiments](docs/site/evals-experiments.md).

Compare a candidate against a baseline and gate CI on regressions with
`compare_experiments()` or `bir eval-gate`. A global `tolerance` bounds how far a
shared evaluator may drop, `score_tolerances` (and repeatable
`--score-tolerance NAME=VALUE`) override that per evaluator, and `missing_score`
(`--missing-score {ignore,regress}`) decides whether an evaluator dropped from
the candidate fails the gate:

```bash
bir eval-gate baseline.jsonl candidate.jsonl \
  --tolerance 0.01 --score-tolerance latency_under=0.05 --missing-score regress
```

The command exits `1` exactly when the policy reports a regression and prints a
machine-readable diff with `effective_tolerances`, `missing_score`, and
`regression_reasons`.

## Documentation

The documentation site covers the [quickstart](docs/site/quickstart.md),
[core API](docs/site/core-api.md),
[capture and privacy](docs/site/capture-privacy.md),
[server uploads](docs/site/sending.md),
[optional integrations](docs/site/integrations.md), and
[local evals and experiments](docs/site/evals-experiments.md).

Build it locally with the isolated documentation extra:

```bash
python -m pip install -e ".[docs]"
mkdocs build --strict
```

CI installs the same `docs` extra and runs the strict build once on every pull
request and push to `main`, so invalid navigation, links reported by MkDocs,
and build warnings block the change.

For local SDK development, install `.[dev]` and see the
[release checklist](docs/SDK_RELEASE_CHECKLIST.md).

```bash
python -m pip install -e ".[dev]" pyright
pyright
python scripts/verify_release.py
```

Release verification builds the wheel without network access from the complete
`bir` package tree, checks its contents and RECORD hashes, then installs it into
a clean virtual environment. The installed-wheel smoke test imports `bir.evals`,
`bir.cli`, and every optional integration module without installing provider
SDKs.

The checked example tests use only standard-library test utilities, so Pyright's
release gate is hermetic whether tooling is installed in a repository `.venv` or
in CI's active interpreter. Pytest remains optional development tooling.

## License

Bir is licensed under the [Apache License 2.0](LICENSE).
