Metadata-Version: 2.5
Name: judgeval
Version: 1.3.2
Summary: The open source post-building layer for Agent Behavior Monitoring.
Project-URL: Homepage, https://github.com/JudgmentLabs/judgeval
Project-URL: Issues, https://github.com/JudgmentLabs/judgeval/issues
Author-email: Andrew Li <andrew@judgmentlabs.ai>, Alex Shan <alex@judgmentlabs.ai>, Joseph Camyre <joseph@judgmentlabs.ai>
Maintainer-email: Judgment Labs <contact@judgmentlabs.ai>
License-Expression: Apache-2.0
License-File: LICENSE.md
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: dotenv>=0.9.9
Requires-Dist: httpx>=0.28.1
Requires-Dist: opentelemetry-exporter-otlp>=1.36.0
Requires-Dist: opentelemetry-sdk>=1.36.0
Requires-Dist: orjson>=3.9.0
Requires-Dist: packaging>=25.0
Requires-Dist: pathspec>=1.0.4
Requires-Dist: typer>=0.9.0
Provides-Extra: s3
Requires-Dist: boto3>=1.40.11; extra == 's3'
Description-Content-Type: text/markdown

<div align="center">

<a href="https://judgmentlabs.ai/">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/logo_darkmode.svg">
    <img src="assets/logo_lightmode.svg" alt="Judgment Logo" width="400" />
  </picture>
</a>

<br>

## The Continuous-Improvement Stack for Agents

Detect failures, triage root causes, and ship fixes backed by production data.

[![PyPI](https://img.shields.io/pypi/v/judgeval?color=orange)](https://pypi.org/project/judgeval/)
[![Docs](https://img.shields.io/badge/Documentation-orange)](https://docs.judgmentlabs.ai/documentation)

[![X](https://img.shields.io/badge/-X/Twitter-000?logo=x&logoColor=white)](https://x.com/JudgmentLabs)
[![LinkedIn](https://custom-icon-badges.demolab.com/badge/LinkedIn%20-0A66C2?logo=linkedin-white&logoColor=fff)](https://www.linkedin.com/company/judgmentlabs)

</div>

## Overview

Judgeval is an open-source Python SDK for agent improvement. It provides tracing and agent-judge evaluation for LLM-powered applications — so you can detect failures, understand what went wrong, and validate fixes against real production cases before shipping.

To get started, dive into the [docs](https://docs.judgmentlabs.ai/documentation).

## Why Judgeval

**OpenTelemetry-based tracing** -- Instrument any function with `@Tracer.observe()`. Automatically captures inputs, outputs, and LLM token usage. Built on OpenTelemetry for full compatibility with existing observability stacks.

**Agent judges** -- Define prompt-based scorers to evaluate agent behaviors at scale. Judges produce structured behaviors — scored, labeled outputs that describe how your agent acted — which accumulate into a searchable record of agent behavior over time. Run judges against live production traffic or replay them on historical traces to validate fixes before shipping.

**Online monitoring** -- Automatically score live production traffic server-side with no latency impact. Detected behaviors surface as structured signals — configure Slack alerts so regressions and recurrences never go unnoticed.

**Broad integrations** -- Auto-instrumentation for OpenAI, Anthropic, Google GenAI, and Together AI. Framework support for LangGraph, OpenLit, and Claude Agent SDK.

## Quickstart

Install the SDK:

```bash
pip install judgeval
```

Set your credentials:

```bash
export JUDGMENT_API_KEY=...
export JUDGMENT_ORG_ID=...
```

Add observability to your agent with two lines of setup:

```python
from judgeval import Tracer, wrap
from openai import OpenAI

Tracer.init(project_name="my-project")
client = wrap(OpenAI())

@Tracer.observe(span_type="tool")
def search(query: str) -> str:
    results = vector_db.search(query)
    return results

@Tracer.observe(span_type="agent")
def run_agent(question: str) -> str:
    context = search(question)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{context}\n\n{question}"}],
    )
    return response.choices[0].message.content

run_agent("What is the capital of the United States?")
```

### JQL

JQL queries use the same API key, organization, and project configuration as the
rest of Judgeval. Tenant identifiers are not part of the query payload.

```python
from judgeval import Judgeval
from judgeval.jql import spans

client = Judgeval(project_name="my-project")
result = client.query(spans().rows(), trace_ids=["trace-123"])
```

`trace_ids` and `session_ids` are mutually exclusive options outside the JQL
query object. Trace IDs narrow the query directly. Judgment resolves session IDs
within the authenticated organization and project, then narrows every part of
the query to their traces. If no session resolves, the request fails instead of
falling back to the whole project. Both options work with `present()` and
`discover()`.

Use `offline_traces()` or `offline_spans()` to query traces captured by an
offline test:

```python
from judgeval.jql import offline_traces

offline_result = client.query(offline_traces().rows())
```

## Integrations

Supports OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, OpenLit, and Claude Agent SDK. See the full [integrations docs](https://docs.judgmentlabs.ai/documentation/integrations/introduction).

## CLI

Manage agents, traces, judges, behaviors, and evaluations from the terminal. Query trace history, deploy judges, inspect detected behaviors, and run evals against production data — all without leaving your shell. See the [CLI repo](https://github.com/JudgmentLabs/cli/) and [docs](https://docs.judgmentlabs.ai/documentation/cli).

## MCP Server

Connect Judgment to any MCP-compatible AI tool. Query agent traces, invoke judges, browse detected behaviors, and surface failures directly inside your AI assistant or IDE. See the [docs](https://docs.judgmentlabs.ai/documentation/mcp-server).

## Links

- [Documentation](https://docs.judgmentlabs.ai/documentation)

---

Judgeval is created and maintained by [Judgment Labs](https://judgmentlabs.ai/).
