RAGOps GitHub

Open source · offline · provider-independent

Catch regressions before production.

Catch regressions in prompts, retrievers, embeddings, datasets, and evaluators before they ship. Compare a candidate with an accepted baseline and get an explainable PASS or BLOCK.

Three-minute proofNo clone · no account · no API key
uvx ragops demo
Read the guide

Creates local JSON, Markdown, and HTML evidence.

Recorded regressionGraph + ACL → lexical only
ACCEPTED BASELINEPASSMeets release policy
CANDIDATEBLOCK3 named gates failed
Versioned policyCompare
Citation coverage−25.00%
Citation precision−25.00%
Lexical groundedness−21.88%
Portable evidenceJSON · Markdown · HTML

One reviewable release loop

From recorded behavior to a decision.

RAGOps stays outside your application. Your model, retriever, and orchestration remain yours; the dependency-free core evaluates portable evidence against a versioned contract.

  1. 01Record

    Answers, citations, retrievals, latency, cost, and metadata.

  2. 02Evaluate

    Deterministic checks, critical findings, and optional external metrics.

  3. 03Compare

    Accepted baseline versus candidate with explicit tolerances.

  4. 04Gate

    Named PASS or BLOCK reasons for people and CI.

Open-source coreScenarios, traces, evaluators, comparison, reports, release gates
Optional adaptersAPI, browser workbench, providers, local control-plane alpha

Stochastic systems need uncertainty

Gate the effect and the uncertainty.

Run the same cases repeatedly, preserve paired case identity, and decide from confidence bounds—not one lucky sample. The statistical path is opt-in; deterministic release semantics stay unchanged.

  1. 01
    Repeat

    Record one metric map for every case and repeat.

  2. 02
    Estimate

    Use a paired hierarchical bootstrap across cases and repeats.

  3. 03
    Gate

    Require both the absolute bound and regression margin.

  4. 04
    Stop

    Inspect only predeclared sequential looks; PASS or BLOCK early.

  5. 05
    Explain

    Separate model, evaluator, dataset, and infrastructure changes.

Fixed-sample comparisonoffline · reproducible seed
ragops compare-runs \
  --baseline-bundle baseline.json \
  --candidate-bundle candidate.json \
  --policy statistical-policy.toml
citation_precisionPASSabsolute bound ≥ 0.90 · delta bound ≥ −0.03
Evaluator driftFrozen anchors test evaluator change inside an equivalence margin.
Accepted baselineContent digests and optional detached SSH signatures bind bundle + policy.
CI integrationReusable GitHub workflow publishes bounded PR evidence.

Run the credential-free fixture →Inspect the reusable GitHub gate →

Evidence, not decoration

Three fixtures. Three bounded claims.

4-case reference deployment

Graph-assisted baseline passes. Lexical-only candidate is blocked.

Citation coverage
−25.00%
Citation precision
−25.00%
Lexical groundedness
−21.88%
Open reference comparison →

30-case synthetic harness benchmark

Baseline passes. Regressed and adversarial candidates are blocked.

Failure families
9
Critical findings
5
Blocked candidates
2
Open benchmark report →

3-case repeated-run acceptance fixture

Fixed comparison passes. Sequential evaluation stops at its first planned look.

Distinct cases
3
Available repeats
2
Stopped at repeat
2
Open statistical fixture →

Evidence boundary: these synthetic results validate the harness, recorded architecture comparison, and statistical acceptance path. They do not establish semantic correctness, production security, customer adoption, causal attribution, or ROI.

Known limits

Every claim keeps its boundary.

01Lexical is not semantic

Lexical groundedness is overlap, not entailment or human judgment.

02Synthetic is not adoption

Fixture results do not prove usage, activation, retention, or ROI.

03Repeats are not new cases

More repeats reduce within-case noise; they never increase distinct-case coverage.

04Fixed bounds are not sequential

Early stopping requires predeclared looks and corrected decision boundaries.

05Provenance is not causality

Changed axes diagnose confounding; they do not prove what caused a regression.

06Local alpha is not SaaS

The control plane is a development surface, not hosted infrastructure.

MIT License · zero hosted fonts · local-first

Make the next RAG change prove it is ready.

Run the demoStar or fork