Dependabot-style automation for LLM workflows

Keep models, prompts, and eval data in sync.

Driftless watches your workflow’s dependencies in CI, repairs prompts when a model or dataset changes, validates through your real eval, and can open an evidence-backed PR when holdout passes.

Run the example Read the docs
Scheduled dependency checks Automatic scoped repairs Holdout-gated pull requests
compare / support-classifier reproducible
$ driftless compare -w support_classifier --to gpt-4o-mini
FROMgpt-4
TOgpt-4o-mini
Same prompt, same eval harness DRIFT FOUND
Macro F1
current1.000
target0.000
Total cost
current0.024
target0.004
Contract checks1 pass  1 fail
✓ max_cost_increase −83.3%
× min_f1 0.000 < 0.900
× target is cheaper, not shippableRepair required →
01Watch dependenciesModels and eval data
02Detect driftDeprecation, cost, quality, labels
03Repair automaticallyPrompt and config in scope
04Propose a PROnly after holdout passes

The dependency problem

Your prompt is the lockfile for a moving system.

Models and eval data are dependencies of the prompt that works today. When either moves, Driftless re-tests the resolved workflow, repairs what is stale, and proposes the update—like Dependabot, but quality-gated.

MODEL CHANGEDExternal drift

Your provider retires a model.

The replacement is cheaper, but your JSON parser starts rejecting fenced output and classification quality drops.

See the model migration use case →
LABELS CHANGEDInternal drift

Your team redefines “correct.”

Support changes its refund policy and updates gold labels. The model stays put, but the prompt now targets yesterday’s rules.

See the label-change use case →

Automatic synchronization

Watch, repair, validate, propose.

Driftless closes the loop from a changed dependency to a reviewable update. Your eval decides what is better, and an untouched holdout decides what can ship.

  1. 01
    Compare

    Run current and target models through the same harness and dataset.

    baseline
  2. 02
    Diagnose

    Cluster schema errors, wrong labels, refusals, and other recurring failures.

    context
  3. 03
    Repair

    Generate candidate edits only inside the files you explicitly allow.

    candidates
  4. 04
    Validate

    Choose on tuning data, then independently verify the winner on holdout.

    gate
DRIFTLESS repair
engine workflow active
failures
patches
scores
final gateHOLDOUT

Your contract

You keep control of the workflow and the quality bar.

One versioned file tells Driftless how to run your app, where the eval lives, what repair may edit, and what “good enough” means.

  • 01
    Bring your own harness

    Driftless shells out to your existing command. Your parsing, retrieval, tools, and post-processing stay intact.

  • 02
    Set a hard edit boundary

    Prompt files, examples, and config can be writable while schemas and product code remain read-only.

  • 03
    Define the merge bar

    Use F1, schema error rate, pass rate, numeric scores, structured fields, or a calibrated LLM judge.

Explore the contract schema →
driftless.yml project / driftless.yml 
version: 1
workflows:
  support_classifier:
    run:
      command: python evals/run_eval.py
      input_path: evals/tickets.jsonl

    model:
      current: gpt-3.5-turbo
      env_var: CLASSIFIER_MODEL

    files:
      editable:
        - prompts/system.md
        - prompts/examples.yml
      readonly:
        - schemas/ticket.schema.json

    thresholds:
      min_f1: 0.90
      max_schema_error_rate: 0.02

    migration:
      holdout_required: true
YAMLUTF-8✓ valid contract

Reviewable outcomes

Every run ends with evidence, not optimism.

Public testbed PR #4LIVE PROOF

Ready for review

The captured testbed run cleared holdout. Reviewers get its scoped prompt diff, score comparison, and attempt history.

Target Macro F10.000 → 0.904
Schema errors100% → 0%
Prompt files changed1 in scope
Blocked migrationISSUE

Nothing unsafe is merged

If no candidate passes holdout, Driftless records what failed and opens an issue instead of creating a false-confidence PR.

attempt_04 FAIL min_f1 0.872 < 0.900 Best candidate retained as evidence
PUBLIC TESTBED PROOF · PR #4 Inspect the real draft PR

A captured testbed run with generated report, untouched holdout result, one prompt repair, and a separate model configuration update in the PR.

Open PR #4 on GitHub →
Public GitHub draft PR created from a passing Driftless migration

Common use cases

Built for real LLM workflow changes.

View all guides →
01Model deprecation

Migrate a classifier without breaking strict JSON or quality.

→ 02Label policy change

Refine the prompt while keeping the model pinned.

→ 03Cost reduction

Adopt a cheaper model only when the same quality bar passes.

→ 04RAG and agents

Test the whole application while keeping retrieval and tools fixed.

→

Try it locally

See the synchronization loop in under a minute.

The bundled demo deterministically shows a model dependency change, the resulting quality regression, and a blocked key-free analysis. It is separate from public testbed PR #4 and does not ship a deterministic repair generator.

✓ No API key ✓ Reproducible fixture ✓ Python 3.10+
Terminal
$ pip install driftless $ driftless copy-example support-classifier --out-dir demo $ cd demo $ driftless validate -w support_classifier $ driftless compare -w support_classifier --to gpt-4o-mini