Metadata-Version: 2.4
Name: tracesweep
Version: 0.0.1
Summary: Reads your agent traces. Tells you what's broken. Doesn't touch your code. (placeholder — not yet built)
Author: Priya Dwivedi
License-Expression: LicenseRef-TBD
Project-URL: Homepage, https://github.com/priya-dwivedi/tracesweep
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# tracesweep

**Reads your agent traces. Tells you what's broken. Doesn't touch your code.**

An open-source alternative to LangSmith Engine: point it at a corpus of production agent traces,
and it screens them, groups failures into named issues, and puts them on a board for a human to
rule on.

A **sweep** is the unit of work — one pass over the corpus. Sweeps are scheduled, compared to each
other, and cost a printed number of dollars.

```bash
pip install tracesweep          # not yet published

tracesweep ingest ./traces
tracesweep run --budget 5.00    # one sweep
tracesweep issues
```

> **Status: pre-alpha. Nothing is built yet.** This repo currently holds the name, the design, and
> the decisions behind it. The blueprint is complete and adversarially reviewed; the code is not
> written.

## Why

LangSmith Engine bills against LangChain-managed inference, and **BYOK is explicitly not
supported** — roughly $750/month at the default org cap. You are paying a vendor's LLM to read your
LLM's output, continuously, at exactly the trace volume where it is most useful.

That pricing is structural, and it is the opening. Aggressive representation tiering plus a cheap
model at the wide end gets the same job done for single-digit dollars a month on your own keys.

## Design in one page

| | |
|---|---|
| **Screening** | Gemini 2.5 Flash-Lite over a bounded ~1,200-token excerpt view, not the full trace. ~95% of spend, ~$3.40 per 50k-trace pass |
| **Investigation** | Claude Agent SDK — subagents, hooks, per-subagent model override. ~4% of spend |
| **Grounding** | Every finding must quote an exact substring from a cited turn, checked by a Python `in` comparison. A model that fabricates a quote is discarded by code, not by another model's opinion |
| **Free detectors** | drain3 error templating and a SQL rule bank run before any model does, at $0 |
| **Fixes** | **None.** It finds and reports. It does not open PRs |

**No auto-PR is a security decision, not a scoping shortcut.** The system ingests production
traces — attacker-controllable end-user text and tool output — into an agent that reads a private
repo. Untrusted input, private data, and external write capability in one loop is the dominant
threat for this product class. Human review breaks the loop.

## What's actually unknown

Nobody publishes precision or false-positive rates for trace issue detection — not LangSmith
Engine, not any competitor. That is the one piece of genuine white space, and it is the point of
building this in the open.

The base-rate arithmetic is unforgiving: at an implied yield of ~1 issue per 3,000 traces, even
99.9% per-trace specificity produces roughly three false issues for every true one. That is
structural, not a tuning problem, and it is what the design is organised around.

## Roadmap

- [ ] **Phase 1** — ingest, compaction, free detectors, screener, a measured recall number on the
      TRAIL benchmark's 841 annotations, with a cost receipt
- [ ] Screener bake-off: Flash-Lite vs Haiku vs GPT-5-nano over TRAIL, recall plotted against cost
- [ ] **Phase 2** — investigator, verifier, issue board, scheduled sweeps
- [ ] Publish precision with a confidence interval — the number nobody else has

## License

TBD — see [#license](https://github.com/priya-dwivedi/tracesweep/issues).
