redline
Local-first prompt regression diffs
Point redline at the prompt-response logs you already have, change your prompt, then catch broken JSON, missing URLs, lost tables, and silent refusals before your users do.
$ python -m pip install redline-ai
$ redline demo --public --compact
$ redline demo --public --compact
redline public dogfood: cases=10 regression=10 changed=0 missing=0 neutral=0
Confidence: HIGH | fix blocking cases before shipping
Scope: structural checks only; review factual correctness, tone, hallucinations, and subtle reasoning separately
REGRESSION case_001: candidate lost valid JSON format
REGRESSION case_002: candidate lost markdown table structure
REGRESSION case_006: candidate newly refuses
REGRESSION case_010: candidate lost bullet list structure
10 blocking regressions - fix before you ship.
$ redline diff dolly-suite-100.json dolly-candidate-100.jsonl --compact
redline diff: cases=100 regression=51 changed=27 missing=0 neutral=22
Diagnosis: candidate got shorter, lost required structure, dropped concrete details, returned empty outputs, and changed content substantially.First-run proof
The public demo is synthetic and safe to share, but it behaves like a real prompt change: the candidate gets shorter and silently drops required structure, dates, owners, URLs, and refusal behavior. The larger internet dogfood run imports 100 rows from Databricks Dolly 15k and pushes them through the same suite, diff, history, benchmark, app, and dashboard surfaces.
baseline
{
"owner": "ana@acme.dev",
"due": "2025-07-01",
"links": ["acme.dev/x"],
"status": "open"
}candidate
owner: ana, due: soon,
links: omitted,
status: open
not valid JSONreason: candidate lost valid JSON format; required keys are no longer parseable.
Imported from Databricks Dolly 15k with redline's JSONL importer and kept as local-only dogfood evidence.
The generated suite covered 100/100 cases and 20/20 deterministic behavior groups.
The report diagnosed shorter answers, lost structure, dropped concrete details, empty outputs, and substantial content drift.
The local app loaded 1 report, 1 benchmark, and 1 history entry with no sidecar-artifact noise.
Product promise
Build suites from JSONL prompt-response logs instead of inventing eval cases by hand.
The core checks run locally with zero runtime dependencies, no API keys, and reproducible CI output.
Reports name the broken behavior: lost JSON, missing keys, dropped URLs, new refusals, empty output, or structure drift.
Mark intentional changes, accept reviewed baselines, and keep the suite evolving with your prompt.
Core loop
Watch an app, import JSONL, or adapt exports from your logging stack.
redline watch --log logs/prompts.jsonl
Select representative behavior from real prompt-output data.
redline suite logs/prompts.jsonl --out redline-suite.json
Replay the suite against a new prompt file or candidate log.
redline eval --prompt prompts/v2.txt
Pin requirements, mark expected changes, and promote reviewed outputs.
redline mark ...
redline accept ...
Trust model
JSON validity, missing keys, URLs, numbers, entities, tables, lists, code blocks, empty outputs, refusals, and obvious policy wording flips are surfaced with reproducible reasons.
Tone, factual accuracy, hallucination risk, and subtle reasoning quality need pinned requirements or an optional judge. redline tells you when a run has no structural blockers; it does not pretend that means the answer is perfect.
Pin edge cases with requirements, add product-specific judge rubrics for ambiguous changes, then accept reviewed candidate outputs so the baseline evolves deliberately.
Bring your stack
Use a command that reads stdin and prints stdout, or copy a runner for OpenAI, Anthropic, LiteLLM, HTTP APIs, LangChain, or LlamaIndex.
redline init --runner stdio --copy-runner
Publish self-contained HTML reports as CI artifacts or browse the ship-readiness dashboard locally.
redline app --reports-dir .redline/demo/reports
AI assistant loop
The MCP server lets Claude, Codex, Cursor, Kiro, and other MCP clients inspect redline setup, run evals, summarize suites, render dashboards, and list cases without switching context.
Use the published MCP entry backed by the PyPI package.
uvx --from redline-ai redline-mcp
Your assistant can run the local check and explain the behavioral diff inline.
Did my prompt change introduce regressions?
Read-only tools are the default; marking cases requires explicit write approval.
redline_mark requires allow_write
Release confidence
redline ships with repeatable checks for the paths that matter: clean install, external project CI behavior, report artifacts, history trends, dashboard output, and release packaging.
The release gate builds a wheel, installs it in a fresh virtualenv, and runs the first-user command path from outside the repo.
bash scripts/release_check.sh
A temporary external project verifies strict doctor, strict validate, eval failure, GitHub annotations, history, and dashboard artifacts.
bash scripts/action_smoke.sh
History shows whether blocking regressions are getting better or worse; summary and doctor surface thin coverage and semantic gaps.
redline history --fail-on worse
Build scripts produce wheel and source distributions and run package metadata checks before anything is uploaded.
bash scripts/certify_release.sh
Public alpha path
The fastest way to understand redline is to make it catch one regression in your own AI workflow.
Open source surface