Apache 2.0 · Open source

Rubra

Every aspect, nothing missed.

A trace-first agent evaluation framework. Decorate your agent once — Rubra captures every tool call, LLM call, token, and cost automatically, then scores it across 36 metrics, including 11 tool-orchestration metrics not commonly found elsewhere.

$ pip install rubra
Get started
PyPI version License Python 3.11+ 
import rubra

# one decorator — that's the entire integration
@rubra.agent(task="Answer questions using web search")
def my_agent(question: str) -> str:
    context = search_web(question)
    return call_llm(context, question)

my_agent("What is the capital of France?")

report = rubra.evaluate(rubra.get_last_trace())
print(f"Rubra Score: {report.rubra_score:.3f}")
# Rubra Score: 0.923
Architecture

How it works

Python contextvars.ContextVar propagates the active trace — async-safe and thread-safe by construction, no globals, no locks. Every @rubra.agent call gets its own isolated trace, so concurrent agents never cross-contaminate.

@rubra.agent wraps your function Trace (ContextVar) @rubra.tool → TOOL_CALL / RESPONSE rubra.patch → LLM_CALL spans evaluate() 36 metrics run EvalReport 36 metrics + 3 composite scores
rubra-sdk

The core library

Zero-config by default — SQLite auto-inits at .rubra/rubra.db. Four core dependencies, no LLM required unless you opt into goal metrics.

Execution

13

Deterministic, zero LLM cost.

task_completion_ratecost_efficiencyretry_rate
Signature metrics

Tool Orchestration

11

Precision/recall/F1 on tool selection, call-order scoring, redundant-call detection.

tool_selection_f1tool_call_order_scoretool_chain_validity

Safety

3

Prompt injection, scope creep, PII propagation.

prompt_injection_resistancepii_propagation_count

Quality

4

Answer relevance, coherence, format, groundedness.

response_groundednessoutput_coherence_score

Goal (LLM-judge)

5

Reference-free, via rubra[judge]. Any litellm model, including free local Ollama.

goal_completionhallucination_score

Composite

3

Weighted roll-ups computed automatically.

rubra_scoretool_intelligence_scoreagentic_efficiency_score
Integrations

Instrument what you already use

OpenAI

Anthropic

LangChain

LangGraph

OpenTelemetry

pytest

↗ PyPI package ↗ Source on GitHub ↗ Changelog
rubra-server

REST API + live dashboard

Self-host a backend for your traces: browse them, run evaluations, and generate HTML reports from a browser — no notebook required.

RubraAgentic Eval Dashboard 1 trace
weather_agent ✓
Report Tokyo weather
0.732Rubra score
0.818Tool intel.
0.600Agentic eff.
MetricScoreResult
task_completion_rate1.000PASS
tool_call_success_rate1.000PASS
latency_score0.000FAIL
cost_efficiency1.000PASS
$ docker run -p 8000:8000 ghcr.io/pm1715/rubra-server

Or pip install rubra-server. Either way, the dashboard polls /api/v1/traces every 30s — traces produced anywhere pointed at the same database show up automatically.

MethodPath
GET/api/v1/traces
POST/api/v1/eval
GET/api/v1/report/{trace_id}
DEL/api/v1/traces/{trace_id}
↗ PyPI package ↗ Docker image (GHCR) ↗ Source on GitHub
rubra-deploy

Kubernetes, via Helm

A production-shaped chart: ClusterIP service, persistent volume for SQLite, liveness/readiness probes against /api/v1/health, and an optional Ingress.

# from the OCI registry — no chart repo needed
helm install rubra oci://ghcr.io/pm1715/charts/rubra \
  --version 0.1.3

# with a real judge-model API key + Postgres:
helm upgrade rubra oci://ghcr.io/pm1715/charts/rubra \
  --set secrets.apiKeySecretName=rubra-api-key \
  --set env.RUBRA_DATABASE_URL="postgresql://..."

Also ships scripts/test-in-kind.sh — builds the real Dockerfile, loads it into a throwaway kind cluster, and does a genuine end-to-end deploy test with no cloud cluster or registry required.

↗ Helm chart (GHCR OCI) ↗ Source on GitHub
Positioning

How Rubra compares

Rubra is early (v0.1.x) and hasn't been battle-tested at the scale TruLens or RAGAS have. An earlier version of this table overstated a few of these rows based on general knowledge rather than checking current docs — the version below has been corrected after actually verifying TruLens's and RAGAS's current capabilities. If something here is still wrong, please open an issue.
FeatureRubraTruLensRAGASDeepEval
Lightweight agent instrumentationYesYes (TruChain/TruGraph, @instrument())Manual (dataset-based)Manual
Tool orchestration metric depth (11 fine-grained metrics)YesPartial (7 agent evaluators)Partial (ToolCallAccuracy/F1)Partial
OpenAI + Anthropic auto-traceYesManualManualManual
Reference-free goal evaluationYesYesPartialPartial
LangGraph + LangChain integrationYesYesNoPartial
Safety metrics (injection, PII, scope)YesNoNoYes
OpenTelemetry exportYesYes (native OTEL — more mature)NoNo
Self-hosted REST API + DashboardYesYes (mature)No (metrics library)Yes
Zero config (SQLite default)YesNoN/APartial
Benchmarked against a public agent datasetNoYes (TRAIL dataset)Not verifiedNot verified

Where Rubra is most confidently different is depth on tool-orchestration mechanics — call-order scoring, redundant-call detection, chain-validity, per-tool latency — which the others expose in narrower form (RAGAS has 2-3 tool-call metrics; TruLens's agent evaluators are broader but don't get this granular). Everywhere else, this is closer to different design choices than Rubra wins. TruLens in particular is a substantially more mature project: natively built on OpenTelemetry, with lightweight one-line instrumentation for LangChain/LangGraph, and its agent evaluators have been benchmarked against a public dataset — something Rubra hasn't done yet.

Author

Built by one person, in the open

Prayansh Mishra

Prayansh Mishra

Creator & maintainer of Rubra

Designed and built end to end — tracer, metrics, server, dashboard, Helm chart, and this page — after hands-on experience evaluating LLM agents surfaced real gaps in existing tools.

↗ GitHub ↗ LinkedIn