v0.1.1 · Apache 2.0 · Open source

Rubra

Every aspect, nothing missed.

A trace-first agent evaluation framework. Decorate your agent once — Rubra captures every tool call, LLM call, token, and cost automatically, then scores it across 36 metrics, including 11 tool-orchestration metrics not commonly found elsewhere.

$ pip install rubra
Get started
PyPI version License Python 3.11+ 
import rubra

# one decorator — that's the entire integration
@rubra.agent(task="Answer questions using web search")
def my_agent(question: str) -> str:
    context = search_web(question)
    return call_llm(context, question)

my_agent("What is the capital of France?")

report = rubra.evaluate(rubra.get_last_trace())
print(f"Rubra Score: {report.rubra_score:.3f}")
# Rubra Score: 0.923
Architecture

How it works

Python contextvars.ContextVar propagates the active trace — async-safe and thread-safe by construction, no globals, no locks. Every @rubra.agent call gets its own isolated trace, so concurrent agents never cross-contaminate.

@rubra.agent wraps your function Trace (ContextVar) @rubra.tool → TOOL_CALL / RESPONSE rubra.patch → LLM_CALL spans evaluate() 36 metrics run EvalReport 36 metrics + 3 composite scores
rubra-sdk

The core library

Zero-config by default — SQLite auto-inits at .rubra/rubra.db. Four core dependencies, no LLM required unless you opt into goal metrics.

Execution

13

Deterministic, zero LLM cost.

task_completion_ratecost_efficiencyretry_rate
Signature metrics

Tool Orchestration

11

Precision/recall/F1 on tool selection, call-order scoring, redundant-call detection.

tool_selection_f1tool_call_order_scoretool_chain_validity

Safety

3

Prompt injection, scope creep, PII propagation.

prompt_injection_resistancepii_propagation_count

Quality

4

Answer relevance, coherence, format, groundedness.

response_groundednessoutput_coherence_score

Goal (LLM-judge)

5

Reference-free, via rubra[judge]. Any litellm model, including free local Ollama.

goal_completionhallucination_score

Composite

3

Weighted roll-ups computed automatically.

rubra_scoretool_intelligence_scoreagentic_efficiency_score
Integrations

Instrument what you already use

OpenAI

Anthropic

LangChain

LangGraph

OpenTelemetry

pytest

↗ PyPI package ↗ Source on GitHub ↗ Changelog
rubra-server

REST API + live dashboard

Self-host a backend for your traces: browse them, run evaluations, and generate HTML reports from a browser — no notebook required.

RubraAgentic Eval Dashboard 1 trace
weather_agent ✓
Report Tokyo weather
0.732Rubra score
0.818Tool intel.
0.600Agentic eff.
MetricScoreResult
task_completion_rate1.000PASS
tool_call_success_rate1.000PASS
latency_score0.000FAIL
cost_efficiency1.000PASS
$ docker run -p 8000:8000 ghcr.io/pm1715/rubra-server

Or pip install rubra-server. Either way, the dashboard polls /api/v1/traces every 30s — traces produced anywhere pointed at the same database show up automatically.

MethodPath
GET/api/v1/traces
POST/api/v1/eval
GET/api/v1/report/{trace_id}
DEL/api/v1/traces/{trace_id}
↗ PyPI package ↗ Docker image (GHCR) ↗ Source on GitHub
rubra-deploy

Kubernetes, via Helm

A production-shaped chart: ClusterIP service, persistent volume for SQLite, liveness/readiness probes against /api/v1/health, and an optional Ingress.

# from the OCI registry — no chart repo needed
helm install rubra oci://ghcr.io/pm1715/charts/rubra \
  --version 0.1.0

# with a real judge-model API key + Postgres:
helm upgrade rubra oci://ghcr.io/pm1715/charts/rubra \
  --set secrets.apiKeySecretName=rubra-api-key \
  --set env.RUBRA_DATABASE_URL="postgresql://..."

Also ships scripts/test-in-kind.sh — builds the real Dockerfile, loads it into a throwaway kind cluster, and does a genuine end-to-end deploy test with no cloud cluster or registry required.

↗ Helm chart (GHCR OCI) ↗ Source on GitHub
Positioning

How Rubra compares

Rubra is early (v0.1.x) and hasn't been battle-tested at the scale TruLens or RAGAS have. This table reflects what each project's public docs and source describe as of this writing, not independent benchmarking — treat it as a starting point for your own evaluation, not a verdict.
FeatureRubraTruLensRAGASDeepEval
1-line agent instrumentationYesNoNoNo
Tool orchestration metrics (11 unique)YesNoNoPartial
OpenAI + Anthropic auto-traceYesManualManualManual
Reference-free goal evaluationYesNoPartialPartial
LangGraph + LangChain integrationYesYesNoPartial
Safety metrics (injection, PII, scope)YesNoNoYes
Self-hosted REST API + DashboardYesYesNoYes
Zero config (SQLite default)YesNoNoPartial

Where Rubra is most confidently different is the 11 tool-orchestration metrics — precision/recall/F1 on tool selection, call-order scoring, redundant-call detection — which the others don't expose as first-class metrics today. Most of the rest of the table is closer to different design choices than better or worse: TruLens and DeepEval in particular have mature ecosystems and production track records Rubra doesn't have yet.

Author

Built by one person, in the open

Prayansh Mishra

Prayansh Mishra

Creator & maintainer of Rubra

Designed and built end to end — tracer, metrics, server, dashboard, Helm chart, and this page — after hands-on experience evaluating LLM agents surfaced real gaps in existing tools.

↗ GitHub ↗ LinkedIn