Metadata-Version: 2.5
Name: tracewright
Version: 0.1.1
Summary: Capture Pydantic AI OpenTelemetry spans as local Pydantic Evals datasets.
Project-URL: Homepage, https://github.com/smigolsmigol/tracewright
Project-URL: Issues, https://github.com/smigolsmigol/tracewright/issues
Author-email: Federico Benini <smigolsmigol@protonmail.com>
License: MIT
License-File: LICENSE
Keywords: opentelemetry,pydantic-ai,pydantic-evals,testing
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.10
Provides-Extra: cache
Requires-Dist: f3dx>=0.0.18; extra == 'cache'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.30; extra == 'dev'
Requires-Dist: pydantic-ai<3,>=2.27; extra == 'dev'
Requires-Dist: pydantic-evals<3,>=2.27; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Provides-Extra: pydantic-ai
Requires-Dist: opentelemetry-sdk>=1.30; extra == 'pydantic-ai'
Requires-Dist: pydantic-ai<3,>=2.27; extra == 'pydantic-ai'
Provides-Extra: pydantic-evals
Requires-Dist: pydantic-evals<3,>=2.27; extra == 'pydantic-evals'
Description-Content-Type: text/markdown

# tracewright

[![CI](https://github.com/smigolsmigol/tracewright/actions/workflows/ci.yml/badge.svg)](https://github.com/smigolsmigol/tracewright/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/tracewright)](https://pypi.org/project/tracewright/)
[![Python](https://img.shields.io/pypi/pyversions/tracewright)](https://pypi.org/project/tracewright/)
[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/smigolsmigol/tracewright/badge)](https://scorecard.dev/viewer/?uri=github.com/smigolsmigol/tracewright)

Capture Pydantic AI OpenTelemetry model spans as local, versionable
[`pydantic-evals`](https://pydantic.dev/docs/ai/evals/) datasets.

Tracewright removes the manual step between observing a useful agent run and
turning it into an eval case. It preserves the prompt, expected output, model,
token usage, timing, and trace identity. Pydantic Evals continues to own dataset
storage, evaluators, execution, and reports.

```text
Pydantic AI run -> OpenTelemetry model span -> Pydantic Evals Case
```

## Quick start

The example uses Pydantic AI's `TestModel`, so it needs no provider key or
hosted observability service.

```bash
pip install "tracewright[pydantic-ai,pydantic-evals]"
```

```python
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings
from pydantic_ai.capabilities import Instrumentation
from pydantic_ai.models.test import TestModel

from tracewright import PydanticAITraceExporter, to_pydantic_evals_dataset

exporter = PydanticAITraceExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))

agent = Agent(
    TestModel(),
    instructions="Answer concisely.",
    capabilities=[Instrumentation(settings=InstrumentationSettings(tracer_provider=provider))],
)
agent.run_sync("What is 2+2?")

dataset = to_pydantic_evals_dataset(exporter.rows, name="agent-regression")
dataset.to_file("agent-regression.yaml")
print(exporter.stats)
```

The resulting file is a normal Pydantic Evals dataset. Review it, commit it
with the code it protects, and evaluate future agent versions against it.

The complete runnable example is
[`examples/pydantic_ai_to_evals.py`](examples/pydantic_ai_to_evals.py).

## Contract

`PydanticAITraceExporter` is a standard OpenTelemetry `SpanExporter`. Attach it
to the tracer provider already used by Pydantic AI. Its `rows` property returns
an immutable snapshot suitable for `to_pydantic_evals_dataset`.

Each replayable model span becomes one `pydantic_evals.Case`:

| Case field | Source |
| --- | --- |
| `inputs` | Last user message |
| `expected_output` | Assistant text output |
| `metadata` | The complete `TraceRow`, including system instructions, model, tokens, timing, and trace IDs |

Non-model spans are ignored. Model spans with missing, redacted, or malformed
content are counted in `exporter.stats` and are never silently promoted into
eval cases.

Content capture is a privacy decision. Pydantic AI instrumentation with
`include_content=False` still produces useful telemetry, but it cannot produce
replayable prompt and output pairs.

## Boundaries

Tracewright deliberately does not:

- replace Pydantic Evals evaluators, execution, or reports;
- replace Logfire trace storage or hosted datasets;
- query the Logfire API or decode OTLP payloads;
- reconstruct tool calls, full multi-turn state, or `SpanTree` objects.

`parse_pydantic_ai_jsonl` remains available for files containing exactly one
serialized span object per line. It is not a generic Logfire export or OTLP
reader. Enriched Tracewright/f3dx JSONL and the original replay CLI remain as
compatibility surfaces.

```bash
tracewright replay traces.jsonl \
  --candidate myapp.replay:my_candidate \
  --candidate-model candidate-v2 \
  --report html=report.html \
  --budget "pass_rate=>=1.0,latency_p95=+10%"
```

New integrations should use `PydanticAITraceExporter` and Pydantic Evals.

## Development

```bash
python -m pip install -e ".[dev]"
ruff check src tests examples
mypy src
pytest -q
python examples/pydantic_ai_to_evals.py
```

CI runs on Linux, macOS, and Windows with Python 3.10 and 3.12. It also
replays Tracewright's serialized `TraceRow` contract across supported Pydantic
versions using Pydantic Canary.

## License

MIT
