Metadata-Version: 2.5
Name: tracelint
Version: 0.8.0
Summary: A linter for agent runs — reads the execution trace and flags structural bugs, deterministically.
Project-URL: Homepage, https://github.com/AshwinUgale/tracelint
Project-URL: Repository, https://github.com/AshwinUgale/tracelint
Project-URL: Issues, https://github.com/AshwinUgale/tracelint/issues
Author: Ashwin Ugale
License: MIT License
        
        Copyright (c) 2026 Ashwin Ugale
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,linter,llm,observability,reliability,static-analysis,tool-calling,traces
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4.0
Provides-Extra: capture
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture'
Provides-Extra: capture-crewai
Requires-Dist: openinference-instrumentation-crewai>=0.1; extra == 'capture-crewai'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-crewai'
Provides-Extra: capture-langchain
Requires-Dist: openinference-instrumentation-langchain>=0.1; extra == 'capture-langchain'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-langchain'
Provides-Extra: capture-smolagents
Requires-Dist: openinference-instrumentation-smolagents>=0.1; extra == 'capture-smolagents'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-smolagents'
Provides-Extra: dev
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: langfuse
Requires-Dist: langfuse>=3.0; extra == 'langfuse'
Provides-Extra: real-agent
Requires-Dist: openai>=1.0; extra == 'real-agent'
Description-Content-Type: text/markdown

# tracelint

> **tracelint checks an agent's execution trace for structural defects, locally or in CI.** It can
> read trace files directly or import traces from your existing observability platform.

It reads what a tool-calling agent *actually did* — the execution trace — and flags structural bugs
deterministically, with the exact evidence and a CI exit code. It runs *after* the run, on the
trace, not on your code, and **no second model ever judges it**. Model-as-judge detection of these
defects is unreliable (published trace-error benchmarks show low localization accuracy); many of
them are *structurally decidable* and need no judge — that is the entire premise of the tool.

**Website: [tracelint.com](https://tracelint.com)** — what it catches, how it works, and how to
wire it into CI.
**[See a live demo report](https://ashwinugale.github.io/tracelint/demo/)** — the constructed
validation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious
cases) plus the robust-vs-buggy recovery scorecard, generated by `tracelint demo`.

## Quick start — capture → lint → CI

The core loop is the whole product:

```
run agent in tests → capture the trace → run tracelint → fail CI on provable defects
```

**1. Capture a trace.** Most people don't have a trace file yet, so `tracelint.capture` records one.
It wraps your framework's stock OpenInference instrumentation, so the captured file lints with no
extra conversion:

```bash
pip install "tracelint[capture-smolagents]"   # or [capture-langchain] / [capture-crewai]
```

```python
from tracelint.capture import capture

with capture("trace.json", framework="smolagents"):
    agent.run("...")            # your agent, unchanged
```

**2. Lint it.**

```bash
tracelint check trace.json --format openinference
```

Exit codes: `0` clean · `2` a structurally-provable defect (`hard_defect`) · `3` an input error.
Heuristic candidates never fail CI on their own; a suppression (a rule that couldn't run) is
disclosed but is not a defect.

**3. Fail CI on a defect.** `check` returns `2` on a provable defect, so it gates a build directly:

```yaml
# .github/workflows/agent-tests.yml
      - run: pip install tracelint
      - run: tracelint check trace.json --format openinference   # exit 2 fails the job
```

See [Add to CI](#add-to-ci) for the ready-made GitHub Action, SARIF code-scanning, and pre-commit.

> The in-process capture helper covers the three in-process frameworks — **smolagents**,
> **LangGraph / LangChain**, and **CrewAI**. Langflow is a running product, not a library you call,
> so it uses the [OTel-to-file recipe](docs/integrations/langflow.md) instead (point its export at a
> file, then `check` it). Already collect traces elsewhere? Any [supported format](#supported-formats)
> works with no capture step.

Prefer to see it work first, with no API key or agent? The keyless demo runs a validation suite and
recovery scorecard end to end:

```bash
pip install tracelint
tracelint demo --html demo.html
```

## Limitations (read first)

1. Deterministic rules catch **structural** defects, not whether the final answer was correct.
2. Hallucinated-argument, loop, and redundant-call findings are **candidates** unless structurally
   proven — legitimate value transforms and intentional retries can trip them; each is shown with
   its evidence for human review, never asserted as a verdict. High-confidence hallucination
   detection requires the tool schema to declare field origins (`x-value-origin`).
3. The recovery scorecard needs labeled task outcomes (success oracles); without them it measures
   **behavioral** recovery only ("did not crash"), a weaker claim than correctness.
4. A trace is only as complete as its instrumentation. A rule whose required field is missing is
   **suppressed with a stated reason** — `tracelint` never lints a partial trace as if complete.

## Supported formats

`check` reads native tracelint JSON by default; `--format` points it straight at the trace your
stack already emits — no manual schema conversion:

```bash
tracelint check trace.json    --format native         # canonical tracelint JSON (default)
tracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL
tracelint check messages.json --format openai          # an OpenAI chat message list
tracelint check trace.json    --format langfuse        # a Langfuse trace export
tracelint check run.json      --format langsmith       # a LangSmith run tree export
```

**OpenTelemetry / [OpenInference](https://github.com/Arize-ai/openinference) is a *format*, not a
platform** — a set of OTel span conventions. So `--format openinference` works from *any*
instrumentation that emits them (Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, datasets like TRAIL)
with no account, no platform, and no vendor lock — and it's the format the capture helper writes.
One shared adapter reaches the whole ecosystem instead of one vendor.

Most rules need no tool schemas, so this works keyless; add `--tools tools.json` to light up the
schema-dependent rules (R1, and R3's high-confidence tier). Don't have one? `tracelint init
spans.json --format openinference -o tools.json` bootstraps a starter contract from the trace —
schemas discovered where the telemetry carries them, behavior fields left as placeholders to review.
A multi-trace input (a `.jsonl` file, a JSON array, or an OTLP export carrying several `trace_id`s)
fans out to one report each. From the library, the same one-liner:

```python
from tracelint import lint_otel_trace

report = lint_otel_trace(spans)   # spans: your OpenInference span export (a list of dicts)
print(report.exit_code)           # 0 or 2
```

## The rules

| Rule | Finding | Tiers |
|------|---------|-------|
| R1 | schema violation — args fail the tool's JSON Schema | `hard_defect` |
| R2a | tool returned an error | `hard_event` (structured signal) / `candidate` (heuristic) |
| R2b | an errored result's value reused by a later side-effecting call | `hard_defect` / `candidate` |
| R3 | hallucinated argument — value not derivable from provenance | `candidate`; `hard_defect` if the field is annotated `provided` |
| R4 | loop — N identical no-progress calls (polls/retries excluded) | `candidate` |
| R5 | redundant call — identical call + identical result, no mutation between | `candidate` |
| R6 | malformed arguments — the emitted tool-call arguments are not valid JSON | `hard_defect` |
| R7 | unknown tool — a call to a tool absent from the declared toolset (possible hallucinated tool) | `candidate` |

`hard_event` and `hard_defect` are orthogonal to the finding kind: a tool-error event is a
`hard_event` from a structured status field but a `candidate` from an exception-like string in
free-form content.

## Input format

A native trace is a JSON object (`.json`, or `.jsonl` for many):

```json
{
  "run_id": "run-1",
  "steps": [
    {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"},
    {"type": "tool_call", "call_id": "c1", "name": "get_order_status", "args": {"order_id": "4521"}},
    {"type": "tool_result", "call_id": "c1", "content": {"status": "processing"}, "status": "ok"},
    {"type": "tool_call", "call_id": "c2", "name": "cancel_order",
     "args": {"order_id": "4521", "reason": "not_shipped"}}
  ],
  "final": "Order 4521 has been cancelled."
}
```

`tools.json` supplies the ground truth the rules check against:

```json
{
  "tools": {
    "cancel_order": {
      "schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                 "required": ["order_id"]},
      "metadata": {"side_effecting": true}
    }
  }
}
```

A tool can also declare **what failure looks like** in its result, so a domain failure returned as
a transport success (HTTP 200 carrying `{"status": "declined"}`) is caught structurally instead of
slipping through:

```json
{
  "tools": {
    "charge_card": {
      "metadata": {
        "side_effecting": true,
        "failure_when": {"pointer": "/status", "in": ["declined", "failed"]}
      }
    }
  }
}
```

`failure_when` is a JSON Pointer into the result plus a match (`in` / `equals` / `exists`); a match
is a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A
side-effecting tool with **no** `failure_when` and an unclassifiable result is *suppressed with a
reason* — never counted as a clean pass.

The rules run against **one canonical trace schema**; a thin **adapter** translates each source's
format into it, so the rules never change: `from_openai_messages`, `from_langfuse_trace`,
`from_langsmith_run`, and `from_otel_spans`. The adapters are validated against live data, not just
the spec — `from_otel_spans` on real [TRAIL](https://huggingface.co/datasets/PatronusAI/TRAIL)
benchmark traces, where tracelint deterministically localized real tool errors, a malformed call,
and excessive-retry loops with no model in the loop. Real exports vary, so a new source may need a
small adapter tweak — and when a field a rule needs is absent, that rule **suppresses** (says so)
rather than guessing, so an unhandled quirk degrades safely.

## Integrations & recipes

tracelint reads the telemetry your stack already emits — one shared adapter reaches the whole
ecosystem. [`docs/integrations/`](docs/integrations/README.md) has short, reproducible one-pagers,
each validated on a **real captured trace**:

- **Frameworks (capture a trace in a test):** [smolagents](docs/integrations/smolagents.md) ·
  [LangGraph / LangChain](docs/integrations/langgraph.md) · [CrewAI](docs/integrations/crewai.md) —
  each captures via the helper above and lints via `--format openinference` with no
  framework-specific code. [Langflow](docs/integrations/langflow.md) is a running product, so it
  uses the OTel-to-file recipe instead of the in-process helper.
- **Platforms (already have traces?):** the always-works baseline is to **export the trace to a file
  and `check` it** —  [Arize Phoenix](docs/integrations/phoenix.md) ·
  [Langfuse](docs/integrations/langfuse.md) · [LangSmith](docs/integrations/langsmith.md) ·
  [OpenLLMetry / Traceloop](docs/integrations/otel.md) ·
  [OpenAI / ShareGPT message lists](docs/integrations/openai.md).

For Langfuse, `pull` is convenience sugar on top of that baseline — it fetches a trace straight to a
file, so `pull` → `check` composes and the file doubles as a saved fixture:

```bash
pip install "tracelint[langfuse]"                 # reads your LANGFUSE_* env vars
tracelint langfuse pull <trace-id> -o trace.json   # writes native tracelint JSON
tracelint check trace.json                          # lint it (native is the default format)
```

These are compatibility validations on real traces, not benchmarks or endorsements.

Straight from a running [Arize Phoenix](https://phoenix.arize.com) instance, from the library:

```python
import phoenix as px
from tracelint import lint_otel_trace

spans = px.Client().get_spans_dataframe().to_dict("records")
print(lint_otel_trace(spans).exit_code)
```

Both Phoenix shapes are handled: the span-export JSON (top-level `span_kind`) and the
`get_spans_dataframe()` records (attributes as `attributes.*` columns).

## Add to CI

`tracelint check` returns exit `2` on a structurally-provable defect, so it gates a build directly.
Point it at the traces your agent test job already produces — a defect fails the job; heuristic
candidates never do.

**GitHub Actions** — the ready-made action:

```yaml
name: lint-agent-traces
on: [push, pull_request]
jobs:
  tracelint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... your step that runs the agent and writes traces to ./traces ...
      - uses: AshwinUgale/tracelint@v0.8.0
        with:
          traces: "traces/*.jsonl"
          format: "openinference"     # or native / openai / langfuse
          tools: "tools.json"          # optional — lights up R1, R3, R2 predicates
```

**Any CI, without the action** — it's one pip install and one command:

```bash
pip install tracelint
tracelint check traces/*.jsonl --format openinference --tools tools.json
```

**GitHub code scanning (SARIF)** — surface findings in the repo's *Security → Code scanning* tab
and as inline pull-request annotations. `--sarif` writes a SARIF 2.1.0 file (mapping `hard_defect`
→ `error`, `hard_event` → `warning`, `candidate` → `note`). It's written *before* the exit-`2`
gate, so an `if: always()` upload step runs even when a defect fails the job:

```yaml
      - run: tracelint check traces/*.jsonl --format openinference --tools tools.json --sarif tracelint.sarif
      - uses: github/codeql-action/upload-sarif@v3
        if: always()          # upload even when tracelint exits 2 on a hard_defect
        with:
          sarif_file: tracelint.sarif
```

**pre-commit** — lint only the trace files a commit touches:

```yaml
repos:
  - repo: https://github.com/AshwinUgale/tracelint
    rev: v0.4.1
    hooks:
      - id: tracelint
        files: ^traces/.*\.jsonl$
        args: ["--format", "openinference", "--tools", "tools.json"]
```

## Library

```python
from tracelint import lint_trace, default_rules, Trace, ToolRegistry

trace = Trace.load("trace.json")
registry = ToolRegistry.load("tools.json")
report = lint_trace(trace, default_rules(), registry)
print(report.exit_code)          # 0 or 2
for f in report.active_findings:
    print(f.rule, f.tier.value, f.summary)
```

See `examples/lint_openinference_phoenix.py` for an offline, keyless end-to-end run (Phoenix-shaped
spans → findings, with and without a tool registry).

## Advanced

These layers sit around the core loop and are opt-in.

**Native test assertion (pytest).** Opt in with `pytest_plugins = ["tracelint.pytest_plugin"]` in
your `conftest.py`; the `trace_capture` fixture then captures a run and lints it inline, so a hard
defect fails the test (candidates never do):

```python
def test_agent(trace_capture):
    with trace_capture(framework="smolagents"):
        agent.run("refund order A100")   # a hard defect in the trace fails this test
```

Pass `assert_clean=False` to inspect `cap.report` yourself instead of auto-failing. It's a thin
wrapper over the capture helper, so the same `capture-<framework>` extra applies.

**Recovery scorecard & fault injection.** Measure how an agent behaves under injected faults, scored
against deterministic success oracles:

```bash
tracelint scorecard --demo --faults timeout,error,rate_limit --runs 5
```

The baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports
a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to
behavioral recovery, labeled as weaker.

**Write findings back into Langfuse.** Beyond reading a trace, tracelint can write its verdict back
into Langfuse's own **Score** model so it appears beside your normal evals. This is the one part
that overlaps with what a platform does itself, so it's an advanced recipe, not the primary path —
the baseline (export or `pull` a trace, then `check` it) always works without it:

```bash
pip install "tracelint[langfuse]"        # v3 SDK; reads your LANGFUSE_* env vars

tracelint langfuse check --trace <trace-id> --tools tools.json               # read-only: prints the plan
tracelint langfuse check --trace <trace-id> --tools tools.json --write-back  # writes Scores
```

With `--write-back`, tracelint writes `tracelint.passed` (BOOLEAN) and `tracelint.hard_defects`
(NUMERIC) at the trace level, plus one score per *certain* finding attached to the exact offending
observation. Heuristic candidates are shown in the terminal but never written back. Scores are keyed
by a stable finding id, so re-running updates them in place rather than duplicating.

## Development

```bash
python -m pytest
ruff check src tests
```

The core is dependency-light (`jsonschema` + stdlib) and the whole test suite is deterministic and
offline. A real OpenAI trace-generating agent lives behind the opt-in `[real-agent]` extra and is
never part of the linter. Python 3.10–3.12.
