When a model or dataset changes, Driftless runs your eval, repairs only the files you allow, and can open a pull request with the evidence — or block the change if quality drops.
You need an eval you can run from the command line. The bundled demo needs no API key. Automatic prompt repair on a real workflow does.
The dependency problem
Models and eval data are dependencies of the prompt that works today. When either moves, Driftless re-tests the resolved workflow, repairs what is stale, and proposes the update—like Dependabot, but quality-gated.
The replacement is cheaper, but your JSON parser starts rejecting fenced output and classification quality drops.
Support changes its refund policy and updates gold labels. The model stays put, but the prompt now targets yesterday’s rules.
Automatic synchronization
Driftless closes the loop from a changed dependency to a reviewable update. Your eval decides what is better, and an untouched holdout decides what can ship.
Run current and target models through the same harness and dataset.
Cluster schema errors, wrong labels, refusals, and other recurring failures.
Generate candidate edits only inside the files you explicitly allow.
Choose on tuning data, then independently verify the winner on holdout.
Your contract
One versioned file tells Driftless how to run your app, where the eval lives, what repair may edit, and what “good enough” means.
Driftless shells out to your existing command. Your parsing, retrieval, tools, and post-processing stay intact.
Prompt files, examples, and config can be writable while schemas and product code remain read-only.
Use F1, schema error rate, pass rate, numeric scores, structured fields, or a calibrated LLM judge.
version: 1
workflows:
support_classifier:
run:
command: python evals/run_eval.py
input_path: evals/tickets.jsonl
model:
current: gpt-3.5-turbo
env_var: CLASSIFIER_MODEL
files:
editable:
- prompts/system.md
- prompts/examples.yml
readonly:
- schemas/ticket.schema.json
thresholds:
min_f1: 0.90
max_schema_error_rate: 0.02
migration:
holdout_required: trueReviewable outcomes
--generator noneREPRODUCIBLE BLOCKThe bundled four-row demo fails the quality gate. Driftless records the blocker and previews an issue instead of a false-confidence PR.
FAIL min_f1 0.000 < 0.900
No prompt files committed
--generator fixtureREPRODUCIBLE PASSThe same example applies a known-good bundled patch, still without provider keys. Reviewers get the prompt diff, scorecard, and holdout checks.
Public draft PR from a 290-label testbed: generated report, untouched holdout, one prompt repair, and a model configuration update. Separate from the four-row demo above.
The published CLI reproduces the four-row pass with --generator fixture. Regenerating PR #4’s exact patch still needs the testbed harness plus --generator llm (or the testbed simulator) and may differ.
Common use cases
Migrate a classifier without breaking strict JSON or quality.
→ 02Label policy changeRefine the prompt while keeping the model pinned.
→ 04Cost reductionAdopt a cheaper model only when the same quality bar passes.
→ 07RAG and agentsTest the whole application while keeping retrieval and tools fixed.
→Try it locally
The bundled demo needs no API key. First compare shows a cheaper target failing the quality gate. --generator none keeps it blocked. --generator fixture applies the known-good patch and can pass. This is separate from public testbed PR #4.
Four-row smoke demo: this proves installation, contract execution, gating, and a key-free pass/block loop. It does not establish production quality, statistical confidence, or provider behavior. Use representative data and --generator llm before shipping a real workflow.
Adopting an existing app? driftless configure <workflow> --apply writes a draft driftless.yml you can review, then creates or appends the real file without rewriting comments. Fill in any leftover placeholders, then run validate. Add CI with init-ci last. Follow the adoption guide →
$ pip install driftless
$ driftless copy-example support-classifier --out-dir demo
$ cd demo
$ driftless validate -w support_classifier
$ driftless compare -w support_classifier --to gpt-4o-mini
$ driftless migrate -w support_classifier --to gpt-4o-mini --generator none
# Expected: BLOCKED with a non-zero exit; continue below.
$ driftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
# Expected: PASS — bundled known-good patch, still no API key.
$ driftless report -w support_classifier
$ driftless open-pr -w support_classifier
# Dry run: no PR or issue is created without --create.
--generator none exits non-zero because the cheaper target fails the quality gate. --generator fixture applies the bundled known-good patch and can pass. Both save evidence for report and the dry-run open-pr preview.