Metadata-Version: 2.4
Name: traceport-agent
Version: 0.1.0
Summary: An open-source improvement loop for production AI agents.
Project-URL: Homepage, https://github.com/TheValmarAI/traceport-agent
Project-URL: Documentation, https://github.com/TheValmarAI/traceport-agent#readme
Project-URL: Issues, https://github.com/TheValmarAI/traceport-agent/issues
Project-URL: Source, https://github.com/TheValmarAI/traceport-agent
Author: Traceport contributors
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: deepagents<0.7,>=0.6.12
Requires-Dist: langchain-openai<2,>=1.1
Requires-Dist: langsmith<0.11,>=0.10.10
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: traceport<0.3,>=0.2.0
Provides-Extra: braintrust
Requires-Dist: traceport[braintrust]<0.3,>=0.2.0; extra == 'braintrust'
Provides-Extra: datadog
Requires-Dist: traceport[datadog]<0.3,>=0.2.0; extra == 'datadog'
Provides-Extra: langfuse
Requires-Dist: traceport[langfuse]<0.3,>=0.2.0; extra == 'langfuse'
Provides-Extra: langsmith
Requires-Dist: traceport[langsmith]<0.3,>=0.2.0; extra == 'langsmith'
Provides-Extra: logfire
Requires-Dist: traceport[logfire]<0.3,>=0.2.0; extra == 'logfire'
Provides-Extra: phoenix
Requires-Dist: traceport[phoenix]<0.3,>=0.2.0; extra == 'phoenix'
Provides-Extra: weave
Requires-Dist: traceport[weave]<0.3,>=0.2.0; extra == 'weave'
Description-Content-Type: text/markdown

# Traceport Agent

Traceport Agent is an open-source, file-based improvement loop for production AI agents. It reads
telemetry through [Traceport](https://github.com/TheValmarAI/traceport), screens compact
sessions, investigates recurring failures with LangChain Deep Agents, and produces reviewable
issues, recurrence evaluators, regression assertions, and coding-agent handoffs.

## How it works

```mermaid
flowchart TD
    J["External scheduler or manual run"] --> F["Fetch sessions through Traceport"]
    C["Watermark + session fingerprints"] --> N{"New or changed session?"}
    F --> N
    N -- "No" --> Z["Finish scan + advance cursor"]
    N -- "Yes" --> V["Build bounded, redacted transcripts<br/>with metrics and objective telemetry facts"]
    V --> B["Group into batches of N<br/>default: 20, concurrency: 4"]
    O["Approved Agent Overview<br/>+ optional agent description"] --> S
    B --> S["Agentic screening<br/>across each bounded session"]
    S --> D{"Candidate failure?"}
    D -- "No" --> Z
    D -- "Yes" --> T["Write all fetched traces once<br/>to a temporary redacted workspace"]
    T --> I["Focused investigation<br/>larger bounded transcript + targeted grep"]
    R["Optional read-only repository"] --> I
    I --> M["Diagnose, cluster, and compare with<br/>open, resolved, and dismissed issues"]
    M --> E{"Evidence threshold met?"}
    E -- "No" --> Z
    E -- "Yes" --> Q["Create, update, or reopen issue"]
    Q --> A["Test recurrence evaluator<br/>against evidence + clean controls"]
    A --> G["Create redacted<br/>regression assertions"]
    G --> H["Optional auditable<br/>coding-agent handoff"]
    H --> Z
```

### Finding categories

Every candidate and durable issue uses one category from this stable taxonomy:

| Category | Use when |
| --- | --- |
| `agent_looping` | The agent repeats actions or tool calls without making useful progress. |
| `context_explosion` | The conversation or working context grows excessively and harms cost, latency, or answer quality. |
| `failed_error_recovery` | The agent responds to an error with an ineffective, repeated, or incorrect recovery. |
| `feature_gap` | A legitimate user request requires functionality the agent does not currently provide. |
| `flawed_plan` | The agent chooses a sequence of steps that is unlikely to achieve the requested outcome. |
| `guardrail_bypass` | The agent performs unsafe or disallowed behavior despite an intended guardrail. |
| `hallucination` | The agent makes a factual or action claim that is not supported by the conversation, tools, or telemetry. |
| `incorrect_tool_arguments` | The agent selects an appropriate tool but supplies incorrect or malformed arguments. |
| `missing_capability_awareness` | A useful capability exists, but the agent fails to recognize or use it. |
| `pii_leak` | The agent exposes personal or sensitive identifying information without appropriate authorization. |
| `response_truncation` | The response ends prematurely because of a length, runtime, or transport cutoff. |
| `silent_tool_error` | A tool fails, but the agent hides the failure or continues as though the call succeeded. |
| `system_prompt_drift` | The agent's behavior materially departs from its configured role or governing instructions. |
| `task_evasion` | The agent avoids or refuses an in-scope task without a justified limitation. |
| `tracing_quality` | Missing or inconsistent telemetry prevents reliable reconstruction of the agent's behavior. |
| `wrong_tool` | The agent chooses an inappropriate tool when a better available tool fits the task. |

### AI agents

| Name | Tools available | Output type | Description |
| --- | --- | --- | --- |
| Session screener | `list_scan_records`, `get_session_transcript` | `CandidateFindingBatch` with findings plus exact `SignalAssessment` records for supplied deterministic signals | Screens bounded batches for agentic failures. It cannot read repository files, hydrate investigator context, expand messages, or assess signal families that were not supplied. |
| Issue investigator | `get_session_context`, `get_message_chunk`, `list_issues`, plus read-only `grep` and targeted file reads under `/telemetry/` and optional `/repo/` | `CandidateFindingBatch` containing zero or one revised finding for the flagged session | Verifies or refutes a screening candidate using larger bounded context. All fetched traces are stored once in a temporary redacted workspace, so it can search across them without another provider request or automatically loading them into its context. |

The Python host orchestrates these two agents directly. There is no supervisor LLM that delegates
or reformats their responses.

Evaluator testing and regression assertion creation are separate host-controlled pipeline steps, not
AI agents. This keeps artifact generation predictable and prevents a model from inventing
executable evaluator logic. Semantic recurrence evaluators receive only trace metrics, compact span
summaries, and bounded beginning/end transcript excerpts. Full span inputs, outputs, raw provider
records, and temporary telemetry files are never dumped into the evaluator prompt.

The first run drafts the Agent Overview and pauses for approval. Later one-shot runs process only
new or changed sessions. Each session is screened once per fingerprint, batches keep model calls
efficient, and only flagged sessions receive the larger investigation context. The loop persists
reviewable artifacts rather than modifying the traced application.

V0.1 is deliberately advisory:

- no trace-provider writes
- no web UI or database
- no repository writes or code execution
- no generated Python or JavaScript evaluators
- no PR creation or deployment
- no external handoff delivery

## Install

Install the provider extra you need:

```bash
uv add 'traceport-agent[langsmith]'
uv add 'traceport-agent[logfire]'
uv add 'traceport-agent[langfuse]'
uv add 'traceport-agent[phoenix]'
```

Python 3.11 or newer is required. Credentials must be supplied through provider and model
environment variables; they are never accepted in `traceport-agent.toml`. The CLI automatically
loads a `.env` file from the current directory. Copy `.env.example` to `.env`; `.env` is ignored by
Git and excluded from package artifacts.

## LangSmith connection example

Install Traceport Agent with the LangSmith adapter:

```bash
uv add 'traceport-agent[langsmith]'
```

Export the LangSmith read credential and the credential for the model Traceport Agent will use.
The values below are placeholders; do not put them in `traceport-agent.toml` or commit them:

```bash
export LANGSMITH_API_KEY="lsv2_..."
export OPENAI_API_KEY="sk-..."
```

To use GPT-5.6 Luna through Pydantic AI Gateway instead of a direct OpenAI key, use the Gateway
token with its OpenAI-compatible chat route:

```bash
export OPENAI_API_KEY="pylf_v..."
export OPENAI_BASE_URL="https://gateway-us.pydantic.dev/proxy/chat/"
```

Then configure the model as `openai:gpt-5.6-luna`. The Gateway's Anthropic proxy route is only for
Anthropic-protocol models; it is not the route used for Luna. See the [Pydantic AI Gateway
integration documentation](https://pydantic.dev/docs/ai/overview/gateway/).

Hosted LangSmith uses its default API endpoint. Set these only when your LangSmith deployment or
account requires them:

```bash
export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_WORKSPACE_ID="your-workspace-id"
```

Before configuring the agent, you can verify the Traceport connection directly. `my-agent` must
be the exact LangSmith project containing the traces you want to analyze:

```bash
traceport traces list \
  --provider langsmith \
  --project my-agent \
  --since 7d \
  --limit 5
```

Then initialize Traceport Agent with the same provider and project:

```bash
traceport-agent init \
  --provider langsmith \
  --project my-agent \
  --repo https://github.com/example/my-agent \
  --model openai:gpt-5.6-luna

traceport-agent doctor
```

This writes the provider and project to `traceport-agent.toml`, while credentials remain in the
process environment. `doctor` checks that the LangSmith adapter, model, state directory, and
optional repository are reachable without printing credential values. `LANGSMITH_PROJECT` is not
required for reading: Traceport Agent passes `[trace].project` from its TOML configuration to
Traceport.

## Optional self-tracing

Traceport Agent can trace its own screening, investigation, evaluator, and issue-lifecycle work to
a separate LangSmith project. This is distinct from the source telemetry that it analyzes:

- **Source telemetry** uses `[trace].project` plus `LANGSMITH_API_KEY`,
  `LANGSMITH_ENDPOINT`, and `LANGSMITH_WORKSPACE_ID`.
- **Traceport Agent telemetry** uses only the four
  `TRACEPORT_AGENT_LANGSMITH_*` variables below.

Set all four Agent variables, or leave all four unset:

```bash
export TRACEPORT_AGENT_LANGSMITH_API_KEY="lsv2_agent_..."
export TRACEPORT_AGENT_LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export TRACEPORT_AGENT_LANGSMITH_WORKSPACE_ID="agent-observability-workspace-id"
export TRACEPORT_AGENT_LANGSMITH_PROJECT="traceport-agent-observability"
```

The Agent tracing client never falls back to the unprefixed source variables. Ordinary
`LANGSMITH_TRACING` or `LANGSMITH_PROJECT` settings also cannot implicitly enable Agent tracing.
The same LangSmith account may be used, but the endpoint, workspace, project, and key remain
explicit. Traceport Agent rejects a self-tracing destination that resolves to the same endpoint,
workspace, and project as its LangSmith source.

When configured, each run appears as a `traceport-agent.scan` trace with child spans for fetching,
concurrent screening, investigation, evaluator work, and issue updates. Agent model calls remain
nested under those operations. Metadata contains operational identifiers and counts, never either
API key. Run `traceport-agent doctor` to see `self-tracing: disabled` or a credential-free
configured status.

## Quick start

```bash
traceport-agent init \
  --provider langsmith \
  --project my-agent \
  --repo https://github.com/example/my-agent \
  --model openai:gpt-5.6-luna

traceport-agent doctor
traceport sessions list --provider langsmith --project my-agent --since 7d --limit 5
traceport sessions show SESSION_ID --provider langsmith --project my-agent
traceport-agent run --verbose
traceport-agent overview show
traceport-agent overview approve
traceport-agent run --verbose
traceport-agent issues list
```

The first run drafts `state/memories/AGENTS.md` and exits. Analysis begins only after the exact
current file hash is approved.

By default the CLI reads `traceport-agent.toml` and stores state under `.traceport-agent/`. Override
them before the command:

```bash
traceport-agent --config /config/traceport-agent.toml --state /state run
```

Run this one-shot command from cron, Kubernetes, Cloud Run Jobs, or another external scheduler.
A six-hour cadence is a useful starting point.

## Commands

```text
traceport-agent init --provider ... --project ... --repo ... --model ...
traceport-agent doctor
traceport-agent overview show
traceport-agent overview approve
traceport-agent run [--since ...] [--until ...] [--max-sessions ...] [--verbose]
traceport-agent issues list|show|close|dismiss|reopen|prioritize|send
traceport-agent evals list|show|test
```

`issues send` invokes `LogDispatcher`. Its result is `logged` or `already_logged`; it makes no
network call and never reports that the handoff was sent.

Use `--verbose` to print safe, immediately flushed progress while a scan is running:

```bash
traceport-agent run --max-sessions 10 --verbose
```

It reports fetch counts, new or unchanged sessions, batch progress, candidate investigations,
issue actions, evaluator testing, regression assertion creation, and the final summary. It never
prints transcript content, model prompts, diagnoses, credentials, or repository contents.

Trace and session inspection belongs to the base Traceport CLI and requires no investigator model
call. Every truncated message preserves both its beginning and end by default:

```bash
traceport traces list --provider langsmith --project my-agent --since 30d
traceport traces show TRACE_ID --provider langsmith --project my-agent
traceport sessions list --provider langsmith --project my-agent --since 30d
traceport sessions show SESSION_ID --provider langsmith --project my-agent
```

Each message has a stable index, original character count, omitted count, and SHA-256 hash.
Expansion is explicit and applies only to terminal output:

```bash
traceport traces show TRACE_ID --provider langsmith --project my-agent --expand 0
traceport traces show TRACE_ID --provider langsmith --project my-agent --expand 0,3
traceport traces show TRACE_ID --provider langsmith --project my-agent --max-chars 5000
traceport traces show TRACE_ID --provider langsmith --project my-agent --full
traceport traces show TRACE_ID --provider langsmith --project my-agent --json
```

Traceport Agent imports the same `traceport.inspection` service directly; it does not shell out to
the CLI. Before model access, the Agent adds privacy redaction and an independent total-context
budget. Context signals use LLM-span token totals rather than copied middleware totals.

## Processing loop

The default scan bootstraps seven days, then advances a watermark with a 15-minute overlap and
session fingerprints derived from underlying traces and feedback. Unchanged sessions are skipped;
sessions that gain traces, change terminal state, or receive new negative feedback are rescanned.
It processes at most 500 sessions, uses batches of 20, and allows four concurrent batches.

Each direct Deep Agent returns a required `CandidateFindingBatch` structured response. Invalid
field names, categories, or value types are returned to that agent for correction before host-side
Pydantic validation.

### Signal coverage

Traceport Agent keeps deterministic checks intentionally small. They only surface explicit
telemetry facts; they do not infer behavioral failures. A fact raises priority but does not
automatically become an issue. The host passes the screener the exact session/signal pairs that
exist. The screener must return exactly one compact `SignalAssessment` per pair: `handled` or
`supports_finding`. Missing, duplicate, and extra assessments are rejected. Sessions with no
deterministic signals return no assessments, while normal agentic screening still runs.

| Signal | Objective telemetry fact | Agentic decision |
| --- | --- | --- |
| Explicit errors | Failed tool spans and failed root traces | Did the agent cause, ignore, or correctly handle the error? |
| Timeouts | Failed spans whose error explicitly says timeout or deadline exceeded | Was recovery missing or incorrect? |
| Online evaluator failures | Failed evaluator spans, failed score/verdict payloads, and LangSmith evaluator feedback | What user-visible failure does the evaluator evidence support? |
| Negative user feedback | Native LangSmith feedback plus provider `feedback_stats`; positive feedback is ignored | Does the conversation support the feedback, and which failure category fits? |

Everything else is agentic. The screener reads the bounded conversation and compares it with the
Agent Overview, configured description, and observed tools. This covers unsupported requests,
refusals, looping, context problems, unusual latency or step counts, and other surprising behavior
without brittle keyword rules. It can classify the result as `feature_gap`,
`missing_capability_awareness`, `task_evasion`, another issue category, or no finding.

External dependency errors are not treated as agent failures when the agent handles them correctly.
Provider feedback is read directly by the Agent integration and does not require changing
Traceport's public trace models. Feedback comments are redacted and bounded before model access.
Scan reports include the number of priority cues and the screener's handled/supports-finding
assessment counts.

The evidence thresholds and target-agent context are configurable:

```toml
[agent]
description = """
Branna helps winery operators manage production tasks and attached operational evidence.
It should acknowledge user-provided material without claiming actions it did not perform.
"""

[issues]
ordinary_min_evidence_sessions = 2
safety_min_evidence_sessions = 1 # PII leaks and guardrail bypasses
existing_min_evidence_sessions = 1

[evaluators]
min_evidence_traces = 2
min_control_traces = 2
min_sensitivity = 0.8
min_specificity = 0.8

[signals]
negative_feedback_max_score = 0.0
evaluator_failure_max_score = 0.0
```

The optional description is included in the screener and investigator context, and in the initial
Agent Overview. Do not put credentials in it. Defaults preserve the two-session ordinary recurrence
rule and the one-session safety exception. Resolved issues reopen on matching evidence; dismissed
issues remain suppressed until a human reopens them.

## Deep Agent access

The directly invoked agents use a `CompositeBackend` with ephemeral scratch state and only the
read-only routes they need:

| Route | Access |
| --- | --- |
| `/skills/` | bundled skills, read-only |
| `/user-skills/` | optional user-controlled skills, read-only |
| `/repo/` | temporary shallow checkout, investigator read-only |

No shell backend is installed. The host creates a credential-free HTTPS GitHub checkout, removes
write permissions, mounts it only for the scan, and never executes repository code.

Typed tool capabilities are separate from methodology skills. Screeners receive only
`list_scan_records` and `get_session_transcript`. Only investigators receive
`get_session_context`, and that tool rejects session IDs that were not explicitly flagged.
Investigators also receive `get_message_chunk` for targeted reads from omitted sections of those
flagged sessions; screeners do not.
The list tool returns metrics and signals without transcript content, preventing duplication when
the screener reads each conversation through the transcript tool.
Screener messages are bounded to 1,000 characters each and 12,000 characters total by default.
Investigator messages are bounded to 4,000
characters each and 30,000 characters total by default; span metadata remains content-free. The
investigator can selectively read redacted chunks from omitted sections. Each chunk is capped at
the per-message limit, and all chunk reads for a session share the investigator total-character
budget. There is no unrestricted full-content tool. Canonical issue, evaluator, memory, and handoff
writes remain host-controlled and are not tools on either analysis agent.

The bundled original skills are:

- `issue-lifecycle`
- `trace-triage`
- `issue-investigation`
- `memory-curation`
- `coding-agent-handoff`

Add user-managed skill directories with `extra_skill_dirs` in the TOML configuration. Traceport
Agent does not vendor LangChain's eval-engineering skill or Phoenix skills.

```toml
extra_skill_dirs = ["/opt/my-agent-skills"]
```

## State and contracts

```text
state/
├── memories/AGENTS.md
├── overview-approval.json
├── cursor.json
├── scans/
├── issues/
├── evaluators/
├── regression-examples/
└── handoffs/
```

Canonical artifacts are schema-validated JSON with `schema_version: 1`. Markdown issue and evaluator
files are regenerated human projections. Writes use a same-directory temporary file, `fsync`, and
atomic replace. An advisory state lock prevents overlapping runs. Each state directory is bound to
one provider/project/repository identity.

Only session IDs, underlying trace IDs, metrics, and capped redacted excerpts are persisted by
default. Full sessions and traces remain in memory during the current run.

Investigator bounds are configurable without changing Traceport's lossless source objects:

```toml
[privacy]
screener_message_chars = 1000
screener_total_chars = 12000
investigator_message_chars = 4000
investigator_total_chars = 30000
extra_secret_patterns = []
```

## Evaluators

Structural evaluators use a whitelisted predicate DSL over trace status, tool counts, repeated
calls, errors, latency, tokens, metadata, contains, and bounded regex. Arbitrary code cannot be
represented or executed. Semantic evaluators require a structured Boolean judge verdict plus an
explanation.

The host derives a recurrence evaluator from the confirmed issue and tests it as its own pipeline
step. It becomes `validated_on_sample` only when it meets all four thresholds from `[evaluators]`;
otherwise it remains `proposed` with its failed cases recorded. Skipped cases do not count, and
tool or model failures remain infrastructure errors.

Regression assertion creation is a separate step and does not depend on evaluator promotion.
Examples retain redacted inputs/context and grounded assertions. Failed production answers are
evidence, never ground truth.

## Container

Build from this package directory:

```bash
docker build -t traceport-agent:0.1.0 .
docker run --rm \
  --env-file traceport-agent.env \
  -v "$PWD/traceport-agent.toml:/config/traceport-agent.toml:ro" \
  -v "$PWD/state:/state" \
  traceport-agent:0.1.0 \
  --config /config/traceport-agent.toml --state /state run
```

See `examples/` for a Kubernetes CronJob and GitHub Actions schedule.

## Development

```bash
uv sync
uv run ruff format --check src tests
uv run ruff check src tests
uv run mypy src
uv run pytest
uv build
```

Live provider and model tests are intentionally opt-in and credential-gated.

Traceport Agent depends on the Traceport `0.2.x` public API and imports its bounded inspection
service directly.

## Methodology and attribution

The compact-screening, specialized-investigation, persistent-memory, and discovery-before-fixing
workflow is informed by [LangSmith Engine's technical
write-up](https://www.langchain.com/blog/how-we-built-langsmith-engine-our-agent-for-improving-agents)
and [Engine documentation](https://docs.langchain.com/langsmith/engine). The tool/skill separation
and telemetry-first diagnostic boundary are also informed by [Arize's agent-harness
article](https://arize.com/blog/closing-the-loop-coding-agents-telemetry-and-the-path-to-self-improving-software/).

Deep Agents is used as an MIT-licensed dependency. The six Traceport Agent skills are original
implementations of public methodology. This package does not copy or vendor LangChain's
eval-engineering skill or Phoenix skills; users may mount separately obtained skills through
`extra_skill_dirs`.
