Metadata-Version: 2.4
Name: orcheval
Version: 0.1.0
Summary: Evaluations for multi-agent orchestrations.
Project-URL: Homepage, https://github.com/doquangg/OrchEval
Project-URL: Repository, https://github.com/doquangg/OrchEval
Project-URL: Issues, https://github.com/doquangg/OrchEval/issues
Author: Quang Do
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.0
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Provides-Extra: langgraph
Requires-Dist: langchain-core>=0.3; extra == 'langgraph'
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.0.7; extra == 'openai-agents'
Provides-Extra: pandas
Requires-Dist: pandas>=1.5; extra == 'pandas'
Description-Content-Type: text/markdown

# OrchEval

[![PyPI](https://img.shields.io/pypi/v/orcheval)](https://pypi.org/project/orcheval/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)

Evaluate, profile, and debug multi-agent LLM systems.

## Why OrchEval
OrchEval arose out of a need to observe a multi-agent class project; I wanted to know
exactly how my system was behaving; when did it reroute, when did it fail, and why? Existing
tools on the market (LangSmith, Langfuse, AgentOps, what have you) require accounts or self-hosted deployments, and I'm ultimately too lazy for that. Besides, some of these services aren't free. 

So anyways, I thought I'd make my own. With a simple `pip install` and a few imports, you can get your orchestration traced and analyzed within a few lines. OrchEval doesn't do any production monitoring or prompt management, but it is quite useful for deep offline analysis of multi-agent behavior that you couldn't see otherwise, such as:

- Why is my agent's prompt growing 120% across retries?
- Is the router oscillating between two nodes instead of converging?
- Did the architecture change actually reduce cost, or just shift it?
- Which of these 50 traces is the outlier, and why?


If you're building with LangGraph, OpenAI Agents SDK, or any other custom orchestration and you want to understand what your agents are actually doing, for free, without signing up for anything, OrchEval is your tool.

### Comparison

| | OrchEval | LangSmith | Langfuse | AgentOps | Arize Phoenix |
|---|---|---|---|---|---|
| **Type** | Python library | Cloud platform | Platform (cloud + self-host) | Cloud platform | Platform (cloud + self-host) |
| **Setup** | `pip install` | Account + API key | Docker or cloud account | Account + API key | Docker or cloud account |
| **Pricing** | Free (Apache 2.0) | Free tier, then $39/seat/mo + per-trace | Free tier, self-host free | Free tier, then $40/mo | Free tier, self-host free (ELv2) |
| **Data stays local** | Always | No (cloud) | Self-host only | No (cloud) | Self-host only |
| **Multi-agent routing analysis** | Built-in (pattern detection) | Manual trace inspection | Manual trace inspection | Session replay | Manual trace inspection |
| **Cross-run aggregation** | Built-in (outliers, trends, shapes) | Dashboard filtering | Dashboard filtering | Dashboard filtering | Dataset comparison |
| **Convergence tracking** | Built-in (per-metric classification) | — | — | — | — |
| **LLM behavioral patterns** | Built-in (5 detectors) | — | — | — | — |
| **Run-to-run diff** | Structured (`compare_runs()`) | Side-by-side in UI | — | — | Experiment comparison |
| **Output format** | Pydantic models, HTML, JSON, Mermaid, DataFrame | Web dashboard | Web dashboard | Web dashboard | Web dashboard + notebooks |
| **CI/CD friendly** | Yes (pure functions, no I/O) | Via API | Via API | Via API | Via API |
| **Framework lock-in** | None | Best with LangChain | None | None | None |

## What OrchEval Analyzes

- Cost and token breakdown by node and model
- Routing decision audit with pattern detection (invariant routing, oscillation, dominant paths)
- Multi-pass convergence tracking with per-metric trend classification
- Retry and error pattern analysis with success rates
- LLM behavioral patterns (prompt growth, stuck agents, redundant tool calls)
- Execution timeline with span hierarchy and state diffs
- Cross-run aggregation with outlier detection and trend analysis

## Architecture

OrchEval has three layers:

```
Collect         →    Report          →    Inspect
─────────────        ──────────────       ──────────────
Adapters emit        report() runs        to_digest()
framework events     6 analysis modules   to_html()
into Traces          compare_runs()       to_mermaid()
                     TraceCollection      to_dataframe()
```

For contributor details, see the directory READMEs:
[`src/orcheval/`](src/orcheval/README.md) |
[`src/orcheval/adapters/`](src/orcheval/adapters/README.md) |
[`src/orcheval/report/`](src/orcheval/report/README.md) |
[`tests/`](tests/README.md)

## Installation

```bash
pip install orcheval                    # core (pydantic only)
pip install orcheval[langgraph]         # + LangGraph adapter
pip install orcheval[openai_agents]     # + OpenAI Agents SDK adapter
pip install orcheval[pandas]            # + DataFrame export
```

## Quickstart

```python
from orcheval import Tracer

tracer = Tracer(adapter="langgraph")
result = graph.invoke(input, config={"callbacks": [tracer.handler]})
trace = tracer.collect()

print(trace.node_sequence())   # ['planner', 'coder', 'reviewer']
print(trace.total_cost())      # 0.025
print(trace.total_tokens())    # {'prompt': 950, 'completion': 350, 'total': 1300}
```

```python
# Compact text digest — feed it to an LLM for analysis
print(trace.to_digest())
```

```
# Trace Digest: a1b2c3d4

## Overview
- **Nodes:** planner → coder → reviewer (3 unique)
- **Duration:** 5300ms
- **Cost:** $0.025 (950 prompt + 350 completion = 1300 tokens)
- **Errors:** 0

## Execution Flow
1. **planner** — 1 LLM call (gpt-4o), 1500ms
2. **coder** — 1 LLM call (gpt-4o), 1 tool call (execute_code), 3000ms
3. **reviewer** — 1 LLM call (gpt-4o-mini), 800ms
```

## Reports

```python
from orcheval import report

full = report(trace)

# Cost breakdown
full.cost.total_cost            # 0.025
full.cost.most_expensive_node   # 'coder'
full.cost.most_expensive_model  # 'gpt-4o'

# Routing audit
full.routing.total_decisions    # 2
full.routing.flags              # [RoutingFlag(flag_type='invariant_routing', ...)]

# Convergence (for multi-pass systems)
full.convergence.is_converging  # True
full.convergence.total_passes   # 3

# Timeline
full.timeline.total_duration_ms # 5300.0

# Retries and errors
full.retries.total_errors       # 0
full.retries.retry_sequences    # []

# LLM behavioral patterns
full.llm_patterns.patterns      # [LLMPattern(pattern_type='prompt_growth', ...)]
```

Individual reports can be generated separately:

```python
from orcheval.report import cost_report, routing_report

cost = cost_report(trace)
routing = routing_report(trace)
```

### Routing Flags

OrchEval detects suspicious routing patterns automatically:

```python
for flag in full.routing.flags:
    print(f"[{flag.flag_type}] {flag.description}")
# [invariant_routing] planner always routes to coder (3/3 decisions)
# [dominant_path] coder routes to reviewer 95%+ of the time
```

Flag types: `invariant_routing`, `context_divergence`, `dominant_path`, `oscillation`.

### LLM Patterns

```python
for p in full.llm_patterns.patterns:
    print(f"[{p.severity}] {p.pattern_type}: {p.description}")
# [warning] prompt_growth: coder input tokens grew 120% across invocations
# [warning] redundant_tool_call: execute_code called 3x with identical input
# [info] output_not_utilized: reviewer LLM output produced but state unchanged
```

Pattern types: `prompt_growth`, `repeated_output`, `redundant_tool_call`, `system_message_variance`, `output_not_utilized`.

## HTML Visualization

```python
trace.to_html("trace.html")  # writes to orcheval_outputs/trace.html
```

Generates a self-contained HTML file (no external dependencies) with:
- Summary metrics panel (duration, cost, tokens, errors)
- Interactive waterfall timeline with swimlane layout per node
- Click-to-expand detail panels showing LLM calls, tool calls, errors, and state diffs

All export methods that accept a bare filename (e.g. `"trace.html"`) write to the `orcheval_outputs/` directory by default. Paths with a directory component or absolute paths are used as-is.

Open `orcheval_outputs/trace.html` in any browser.


https://github.com/user-attachments/assets/04f69cc1-c911-4259-b108-fc323852031c


## Text Digest

```python
# Compact overview
print(trace.to_digest())

# Focus on specific nodes, collapse others into a summary line
print(trace.to_digest(focus_nodes=["coder"]))

# Include full LLM prompt/response content
print(trace.to_digest(include_llm_content=True))

# Control output size (~4 chars per token)
print(trace.to_digest(max_chars=8_000))

# Reuse a precomputed FullReport
print(trace.to_digest(reports=full))
```

## Run Comparison

```python
from orcheval import compare_runs

diff = compare_runs(baseline_trace, experiment_trace)

# Natural-language summary of all changes
print(diff.summary)
# "Cost increased $0.025 → $0.031 (+24.0%). coder duration flagged: 3000ms → 4200ms (+40.0%)."

# Programmatic access
diff.cost.total_delta.delta       # 0.006
diff.cost.total_delta.pct_change  # 24.0
diff.duration.total_delta.flagged # True
diff.errors.new_errors            # []
diff.llm_patterns.new_patterns    # [PatternDiff(...)]

# Or compare directly from a trace
diff = baseline_trace.compare(experiment_trace)
```

## Cross-Run Aggregation

```python
from orcheval import TraceCollection

collection = TraceCollection.from_traces(trace1, trace2, trace3, trace4, trace5)
# Or load from a directory of JSON files:
# collection = TraceCollection.from_json_dir("traces/")

# Aggregate statistics
summary = collection.summary()
summary.trace_count                # 5
summary.total_cost.mean            # 0.027
summary.total_cost.p95             # 0.035
summary.unique_node_names          # ['planner', 'coder', 'reviewer']

# Per-node breakdown
coder_stats = collection.node_stats("coder")
coder_stats.duration.median        # 3100.0
coder_stats.cost.mean              # 0.015
coder_stats.error_rate             # 0.2

# Outlier detection
outliers = collection.find_outliers("cost", threshold=2.0)
for o in outliers:
    print(f"Trace {o.trace_id}: {o.metric}={o.value:.3f} (median={o.median:.3f}) — {o.reason}")
# Trace abc123: cost=0.052 (median=0.027) — cost is 1.93x the median (threshold: 2.0x)

# Execution shape clustering
for shape in collection.execution_shapes():
    print(f"{shape.node_sequence} — {shape.trace_count} traces ({shape.fraction:.0%})")
# ['planner', 'coder', 'reviewer'] — 4 traces (80%)
# ['planner', 'coder', 'coder', 'reviewer'] — 1 trace (20%)

# Trend analysis
trend = collection.trend("cost")
trend.direction   # 'increasing'
trend.change_pct  # 15.2
```

## Mermaid Export

```python
print(trace.to_mermaid())
```

```mermaid
graph LR
    planner["planner (1x)"]
    coder["coder (1x)"]
    reviewer["reviewer (1x)"]
    planner -->|"1x"| coder
    coder -->|"1x"| reviewer
```

GitHub renders Mermaid blocks natively. Nodes with errors are highlighted in red.

## DataFrame Export

```python
df = trace.to_dataframe()  # requires pip install orcheval[pandas]
```

One row per event. Columns include: `event_type`, `timestamp`, `node_name`, `span_id`, `duration_ms`, `model`, `input_tokens`, `output_tokens`, `cost`, `tool_name`, `error_type`, `source_node`, `target_node`, and more.

## Framework Support

### LangGraph

```python
from orcheval import Tracer

tracer = Tracer(adapter="langgraph")
result = graph.invoke(input, config={"callbacks": [tracer.handler]})
trace = tracer.collect()
```

Options:

```python
# Infer routing decisions from node transitions
tracer = Tracer(adapter="langgraph", infer_routing=True)

# Capture input/output state on each node
tracer = Tracer(adapter="langgraph", capture_state=True)
```

### OpenAI Agents SDK

#### Note: The OpenAI Agents SDK adapter is experimental and has not been validated against real workloads. If you encounter issues, please open an issue or use the ManualAdapter as a fallback.

```python
from orcheval import Tracer
from agents.tracing import add_trace_processor

tracer = Tracer(adapter="openai_agents")
add_trace_processor(tracer.handler)

result = await Runner.run(agent, "Summarize the document")
trace = tracer.collect()
```

Options:

```python
# Infer routing decisions between agents
tracer = Tracer(adapter="openai_agents", infer_routing=True)

# Capture agent metadata (name, tools, handoffs, output_type)
tracer = Tracer(adapter="openai_agents", capture_state=True)
```

### Manual Adapter

For frameworks without a built-in adapter:

```python
from orcheval import Tracer

tracer = Tracer()  # defaults to manual
a = tracer.adapter

a.node_entry("agent")
a.llm_call(node_name="agent", model="gpt-4o", input_tokens=150, output_tokens=80, cost=0.005)
a.tool_call("search", node_name="agent", tool_input={"query": "test"}, tool_output="result")
a.node_exit("agent", duration_ms=3000.0)

trace = tracer.collect()
```

Same `Trace` object, same analysis, regardless of framework.

## Saving and Loading Traces

```python
# Serialize
json_str = trace.to_json()          # returns JSON string
trace.to_json("trace.json")         # also writes to orcheval_outputs/trace.json
d = trace.to_dict()

# Deserialize
from orcheval import Trace
loaded = Trace.from_json(json_str)          # from string
loaded = Trace.from_json_file("trace.json") # from file
loaded = Trace.from_dict(d)

# Merge multiple traces
combined = Trace.merge(trace1, trace2, trace3)
```

### Generate HTML from saved files

```python
from orcheval import html_from_files

# From trace file only (report auto-generated)
html_from_files("orcheval_outputs/trace.json", output_path="trace.html")

# From trace + pre-computed report
html_from_files("orcheval_outputs/trace.json", "report.json", output_path="trace.html")

# Or load individually
from orcheval import Trace, FullReport
trace = Trace.from_json_file("orcheval_outputs/trace.json")
report = FullReport.from_json_file("report.json")
trace.to_html("trace.html", reports=report)
```

## Contributing

Contributions are welcome. OrchEval follows a standard open-source workflow:

1. **Open an issue first.** Whether it's a bug report, feature request, or a question — start with a [GitHub issue](https://github.com/doquangg/OrchEval/issues). This lets us discuss the approach before you write code.

2. **Fork and branch.** Fork the repo, create a branch from `main`, and make your changes. Keep commits focused — one logical change per PR.

3. **Follow existing conventions.** The codebase uses `ruff` for linting and formatting, `mypy` for type checking, and `pytest` for tests. Run all three before submitting:
   ```bash
   pip install -e ".[dev]"
   ruff check src/ tests/
   mypy src/
   pytest
   ```

4. **Write tests.** If you're adding a feature, add tests. If you're fixing a bug, add a test that reproduces it. Use the `ManualAdapter` and `_ts()` helpers from `tests/conftest.py` — no framework dependencies or API keys needed.

5. **Submit a PR.** Reference the issue number, describe what changed and why, and keep the diff reviewable. Large changes should be discussed in the issue first.

### Adding a new adapter

See [`src/orcheval/adapters/README.md`](src/orcheval/adapters/README.md) for the full guide. The short version: subclass `BaseAdapter`, implement `get_callback_handler()`, register in `Tracer._resolve_adapter()`, and add an optional-dependency group in `pyproject.toml`.

### Adding a new report module

See [`src/orcheval/report/README.md`](src/orcheval/report/README.md). Write a pure function `your_report(trace: Trace) -> YourReport` with frozen Pydantic output models, add it to `FullReport`, and export from `__init__.py`.

## Known Limitations

- **LangGraph routing detection**: LangGraph does not provide explicit routing/conditional-edge
  callbacks. Pass `infer_routing=True` to `Tracer` to emit inferred `RoutingDecision` events
  based on node transition sequences. These carry `metadata={"inferred": True}` and may not
  reflect actual conditional logic. For precise routing data, use the manual adapter's
  `routing_decision()` method.

- **Pass boundaries / convergence tracking**: Neither the LangGraph adapter nor the OpenAI Agents
  adapter emits `PassBoundary` events automatically. `convergence_report()` will always return an
  empty report unless you record pass boundaries manually via `ManualAdapter.pass_boundary()`.
  This means convergence analysis requires explicit user instrumentation regardless of framework.

- **Cost data**: `LLMCall.cost` is a passthrough field — it is auto-populated when the LLM
  provider reports cost in the callback response. OrchEval does not include a built-in pricing
  table. When cost is unavailable, the cost report falls back to token counts and call counts.

## License

Apache 2.0
