Metadata-Version: 2.5
Name: rubra
Version: 0.1.2
Summary: Rubra — Agentic evaluation framework. Every aspect, nothing missed.
Project-URL: Homepage, https://github.com/pm1715/rubra-sdk
Project-URL: Issues, https://github.com/pm1715/rubra-sdk/issues
Project-URL: Changelog, https://github.com/pm1715/rubra-sdk/blob/main/CHANGELOG.md
Author-email: Prayansh Mishra <iprayanshmishra15@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: agent,agentic,ai,evals,evaluation,llm,observability,tracing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.9
Provides-Extra: all
Requires-Dist: anthropic>=0.20; extra == 'all'
Requires-Dist: langchain-core>=0.1; extra == 'all'
Requires-Dist: langchain>=0.1; extra == 'all'
Requires-Dist: langgraph>=0.2; extra == 'all'
Requires-Dist: litellm>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: opentelemetry-api>=1.20; extra == 'all'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20; extra == 'all'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.20; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: litellm>=1.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: openai>=1.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: judge
Requires-Dist: litellm>=1.0; extra == 'judge'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.1; extra == 'langchain'
Requires-Dist: langchain>=0.1; extra == 'langchain'
Provides-Extra: langgraph
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.20; extra == 'otel'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/wordmark.svg" alt="Rubra" width="380"/>
</p>

**Agentic evaluation framework. Every aspect, nothing missed.**

[![PyPI](https://img.shields.io/pypi/v/rubra.svg)](https://pypi.org/project/rubra/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](pyproject.toml)
[![CI](https://github.com/pm1715/rubra-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/pm1715/rubra-sdk/actions)

Rubra is a **trace-first** agent evaluation framework. Decorate your agent — Rubra automatically captures every tool call, LLM call, token, and cost. Then evaluate with 36 metrics, including 11 tool-orchestration metrics not commonly found elsewhere.

```python
import rubra

@rubra.agent(task="Answer questions using web search")
def my_agent(question: str) -> str:
    context = search_web(question)
    return call_llm(context, question)

my_agent("What is the capital of France?")

report = rubra.evaluate(rubra.get_last_trace())
print(f"Rubra Score: {report.rubra_score:.3f}")   # 0.923
print(f"Passed:      {report.passed}/{report.total_metrics}")
```

---

## How Rubra compares

Rubra is early (v0.1.x) and hasn't been battle-tested at the scale TruLens or RAGAS have — the table below reflects what each project's public docs and source describe as of this writing, not independent benchmarking. Treat it as a starting point for your own evaluation, not a verdict.

| Feature | **Rubra** | TruLens | RAGAS | DeepEval |
|---------|:---------:|:-------:|:-----:|:--------:|
| 1-line agent instrumentation | ✅ | ❌ | ❌ | ❌ |
| Tool orchestration metrics (11 unique) | ✅ | ❌ | ❌ | Partial |
| OpenAI + Anthropic auto-trace | ✅ | Manual | Manual | Manual |
| Reference-free goal evaluation | ✅ | ❌ | Partial | Partial |
| LangGraph + LangChain integration | ✅ | ✅ | ❌ | Partial |
| Safety metrics (injection, PII, scope) | ✅ | ❌ | ❌ | ✅ |
| OpenTelemetry export | ✅ | ❌ | ❌ | ❌ |
| Self-hosted REST API + Dashboard | ✅ | ✅ | ❌ | ✅ |
| Pytest plugin | ✅ | ❌ | ❌ | ✅ |
| Zero config (SQLite default) | ✅ | ❌ | ❌ | Partial |

Where Rubra is most confidently different is the **11 tool-orchestration metrics** — precision/recall/F1 on tool selection, call-order scoring, redundant-call detection — which the others don't expose as first-class metrics today. Most of the rest of the table is closer to "different design choices" than "better or worse": TruLens and DeepEval in particular have mature ecosystems and production track records Rubra doesn't have yet.

---

## Installation

```bash
pip install rubra                    # core (4 deps, no LLM required)
pip install "rubra[judge]"           # + LLM-judge metrics via litellm
pip install "rubra[openai]"          # + OpenAI SDK interceptor
pip install "rubra[anthropic]"       # + Anthropic Claude interceptor
pip install "rubra[langgraph]"       # + LangGraph node tracing
pip install "rubra[langchain]"       # + LangChain callback handler
pip install "rubra[otel]"            # + OpenTelemetry export
pip install "rubra[all]"             # everything
```

---

## Quickstart

### 1. Basic agent (any framework)

```python
import rubra

@rubra.tool
def search_web(query: str) -> str:
    return my_search_api(query)

@rubra.agent(
    task="Answer capital city questions",
    expected_tool_calls=["search_web"],   # optional: enables F1 metrics
)
def capital_agent(question: str) -> str:
    context = search_web(question)
    return my_llm(context, question)

capital_agent("What is the capital of Japan?")

trace = rubra.get_last_trace()
report = rubra.evaluate(trace, metrics="all")
print(report.summary())
```

### 2. With OpenAI — zero-change LLM tracing

```python
import openai
import rubra

client = rubra.patch(openai.OpenAI())   # one line — that's it

@rubra.agent(task="Capital cities")
def agent(q: str) -> str:
    response = client.chat.completions.create(   # automatically traced
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": q}],
    )
    return response.choices[0].message.content
```

### 3. With Anthropic Claude

```python
import anthropic
import rubra

client = rubra.patch_anthropic(anthropic.Anthropic())

@rubra.agent(task="Summarise documents")
def agent(text: str) -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{"role": "user", "content": text}],
    )
    return response.content[0].text
```

### 4. In pytest — evaluate your agent in CI

```python
# test_agent.py  (no conftest.py needed — plugin registers automatically)

def test_capital_agent_quality(rubra_trace):
    result = capital_agent("What is the capital of Japan?")
    assert result == "Tokyo"

    report = rubra_trace.evaluate(metrics="execution")
    assert report.get("task_completion_rate").passed
    assert report.rubra_score >= 0.70

# One-liner shorthand:
def test_passes_score_threshold(rubra_trace):
    capital_agent("What is the capital of Germany?")
    rubra_trace.assert_score(min_rubra_score=0.70, min_pass_rate=0.80)
```

### 5. LangGraph

```python
from langgraph.graph import StateGraph
from rubra.integrations.langgraph import patch
import rubra

graph = StateGraph(MyState)
graph.add_node("search", search_node)
graph.add_node("answer", answer_node)
app = patch(graph).compile()   # wraps every node as a tool span

@rubra.agent(task="Multi-hop question answering")
def run(question: str) -> str:
    return app.invoke({"question": question})["answer"]
```

### 6. LangChain

```python
from rubra.integrations.langchain import RubraCallbackHandler
import rubra

handler = RubraCallbackHandler()

@rubra.agent(task="Chain execution")
def run(question: str) -> str:
    return my_chain.invoke({"question": question}, config={"callbacks": [handler]})
```

---

## Available Metrics

### Execution (13) — deterministic, no LLM needed
| Metric | Description |
|--------|-------------|
| `task_completion_rate` | Did the agent reach COMPLETED status? |
| `tool_call_success_rate` | Fraction of tool calls with no error |
| `error_rate` | 1 − (error spans / total spans) |
| `step_efficiency` | Penalty for exceeding max_steps |
| `latency_score` | Penalty for slow traces |
| `token_efficiency` | Penalty for excess token usage |
| `cost_efficiency` | Linear decay past budget |
| `tool_diversity` | Unique tools / total calls |
| `retry_rate` | Same-tool-after-error retries |
| `hallucination_free_calls` | Empty-argument proxy |
| `response_completeness` | Final output length check |
| `tool_output_utilization` | Tool output present in final response |
| `execution_time_distribution` | Dominant span fraction check |

### Tool Orchestration (11) — USP, unique to Rubra
| Metric | Description |
|--------|-------------|
| `tool_selection_precision` | TP / (TP + FP) vs expected tool calls |
| `tool_selection_recall` | TP / (TP + FN) |
| `tool_selection_f1` | Harmonic mean of precision + recall |
| `tool_call_order_score` | LCS-based sequence alignment |
| `tool_trajectory_equivalence` | Jaccard + order for non-deterministic paths |
| `redundant_tool_call_rate` | Same tool + args called twice |
| `tool_error_recovery_rate` | Does agent continue after tool failure? |
| `intermediate_step_grounding` | Next-call args reference prior response |
| `tool_argument_completeness` | All argument values non-empty |
| `tool_response_latency_score` | Per-tool latency check |
| `tool_chain_validity` | Every TOOL_CALL has a matching TOOL_RESPONSE |

### Safety (3)
`prompt_injection_resistance` · `scope_creep_score` · `pii_propagation_count`

### Quality (4)
`answer_relevance_proxy` · `output_coherence_score` · `format_compliance_score` · `response_groundedness`

### Goal / LLM-judge (5) — requires `rubra[judge]`
`goal_completion` · `answer_correctness` · `reasoning_quality` · `task_understanding` · `hallucination_score`

The judge model is configurable and works with any [litellm](https://docs.litellm.ai/docs/providers)-supported model — including free local models via Ollama, so you can exercise these metrics with zero API cost:

```python
report = rubra.evaluate(trace, metrics="all", judge_model="ollama/llama3.2")
```

### Composite scores (automatic)
- **`rubra_score`** — weighted average across all scored metrics
- **`tool_intelligence_score`** — average of tool-category metrics
- **`agentic_efficiency_score`** — completion × average efficiency

---

## REST API + Dashboard

See [rubra-server](https://github.com/pm1715/rubra-server) for the self-hosted FastAPI backend and live dashboard.

```bash
git clone https://github.com/pm1715/rubra-server
cd rubra-server
docker compose up
# Dashboard → http://localhost:8000
# API docs  → http://localhost:8000/docs
```

---

## CLI

```bash
rubra traces              # list recent traces
rubra eval                # evaluate latest trace
rubra eval <TRACE_ID>     # evaluate specific trace
rubra report -o out.html  # generate HTML report
```

---

## Architecture

Rubra uses Python `contextvars.ContextVar` for async-safe, thread-safe trace propagation — no globals, no thread-locals, no locks. Each `@rubra.agent` call creates an isolated `Trace` with its own `ContextVar` token, making concurrent agents safe by design.

```
@rubra.agent ──► Trace (ContextVar)
    @rubra.tool ──► TOOL_CALL + TOOL_RESPONSE spans
    rubra.patch ──► LLM_CALL spans (auto)
evaluate(trace) ──► EvalReport (36 metrics + 3 composite scores)
```

---

## Author

Rubra was designed and built by **Prayansh Mishra** ([@pm1715](https://github.com/pm1715) · [LinkedIn](https://www.linkedin.com/in/prayansh-mishra-02a57724b/)).

## License

Apache 2.0 — see [LICENSE](LICENSE).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).
