Keep every LLM workflow in sync.

Public alpha

Driftless is published for technical early adopters who already have an executable LLM eval. The bundled examples are key-free (blocked with --generator none, passing with --generator fixture), and the CLI plus composite Action support real local and CI adoption. Integrating your own workflow requires a contract, a reliable eval harness, and deliberate provider-cost and secret handling. Hosted bot/SaaS features are out of scope; see GA criteria.

Introduction

LLM providers deprecate and retire models on aggressive timelines. When a model is retired, every workflow that depends on it breaks or silently degrades. Swapping a model ID is rarely enough — the new model behaves differently, so prompts, format instructions, and few-shot examples often need to change too, and you need evidence that the new model still meets your quality bar before you ship.

driftless is CLI-first: the CLI is the engine, and the GitHub Action just invokes the same commands. You describe your model-dependent workflow once in a driftless.yml contract, and the tool orchestrates everything else.

Installation

Install the published CLI from PyPI:

copypip install driftless

For local development from source:

copypython -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

The bundled examples, --generator none, and --generator fixture paths need no provider key. For LLM-backed repair, install the optional extra and export a key:

copypip install "driftless[llm]"      # openai + anthropic SDKs
export OPENAI_API_KEY=sk-...   # or ANTHROPIC_API_KEY

Quickstart

Start with the bundled support classifier. It is deterministic and runs without provider keys:

copypip install driftless
driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini

The comparison intentionally shows F1 1.000 → 0.000 and total cost 0.024 → 0.004. The cheaper target is unsafe, so FAIL min_f1: 0.000 >= 0.9 is the expected result.

Continue through the key-free blocked path, or reproduce a passing bundled repair:

copydriftless migrate -w support_classifier --to gpt-4o-mini --generator none
driftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
driftless report -w support_classifier
driftless open-pr -w support_classifier

--generator none exits non-zero with BLOCKED. --generator fixture applies the known-good bundled patch and can PASS. Neither calls a provider. open-pr previews the issue or PR without creating it unless you pass --create.

All bundled projects can be copied with copy-example: support-classifier, rag-qa, and tool-agent.

Adopt Driftless in an existing repository

Prove the local path before adding automation. If you are unsure which path fits your goal, start with the command chooser.

1. Discover and configure: scan for model usage, then turn a detected workflow into a contract.
2. Validate locally: make sure the harness runs and compare one target model.
3. Set boundaries: review edit scope, thresholds, known limits, and provider budget.
4. Add CI last: generate workflows with init-ci, review the files, then commit only the jobs you want.

copycd your-existing-repo
driftless scan
driftless configure <workflow> --apply
# Review the inferred contract; resolve any remaining placeholders.
driftless validate -w <workflow>
driftless compare -w <workflow> --to <model>
driftless init-ci

If discovery does not fit your setup, driftless init writes a commented driftless.yml template. The guided repository walkthrough includes a concrete draft-to-contract before/after, provider-cost guidance, and a staged safety checklist.

The workflow contract

A driftless.yml declares one or more model-dependent workflows. Each is a strict, typed schema — unknown keys are rejected, so typos surface as errors rather than silent misbehavior.

copyworkflows:
  support_classifier:
    run:
      command: "python evals/run_eval.py"
      input_path: evals/inputs.jsonl
      output_path: evals/outputs.jsonl
      timeout_seconds: 600
    model:
      current: gpt-3.5-turbo
      target_candidates: [gpt-4o-mini]
      env_var: SUPPORT_CLASSIFIER_MODEL
      config_file: config/llm.yml
      config_path: model
    files:
      editable: [prompts/system.md, prompts/examples.yml]
      context: [src/parser.py]
      readonly: [src/**]
    eval:
      labels_path: evals/labels.jsonl
      schema_path: schemas/ticket.schema.json
      label_field: category
      split: { tuning: 60%, seed: 7 }
    thresholds:
      min_f1: 0.9
      max_schema_error_rate: 0.02
    migration:
      max_iterations: 4
      holdout_required: true
      split_seed_count: 1

There are two override mechanisms, and a workflow may use both:

• env_var — the harness sets this environment variable to run the workflow under a different model. This is the runtime override used during evaluation.
• config_file + config_path — a dotted path into a JSON/YAML config file. Used when opening a PR, to write the new model ID into the repo.

Percentages are ergonomic: 60%, 60, and 0.6 all parse to 0.6.

Command chooser

• Try it without keys: copy-example support-classifier, then follow the blocked quickstart.
• Check setup: validate. Measure a model swap: compare.
• Repair for a model change: migrate. Re-optimize after data changes: refine.
• Triage in CI: plan. Preview the resulting PR/issue: open-pr without --create.

See the complete command chooser for label audits, judge checks, and automation variants.

The migration loop

This is the core of the product. The same loop powers migrate and refine; they differ only in the objective:

• MEET_THRESHOLDS (migrate) — a new model regressed; repair until the contract's thresholds: pass. Candidates are ranked by how close they are to passing.
• MAXIMIZE (refine) — the dataset changed, so the old thresholds are stale; the model is pinned, the loop chases the best primary metric, then proposes fresh thresholds.

Full algorithm, from run_migration in engine.py (primary = F1 or score/pass-rate; diff_size = changed lines vs. the original editable files; evaluate = apply files in a backup/restore sandbox, run your real workflow, score the split):

copyrun_migration(W, M_target, generator G, objective O, seed):

  # ── Setup ──────────────────────────────────────────────
  tuning, holdout ← split(W.dataset, seed)        # deterministic, seeded
  baseline     ← evaluate(M_current, current_files, tuning)
  naive_target ← evaluate(M_target,  current_files, tuning)

  # ── Short-circuit: is a bare model swap enough? (migrate) ─
  if O = MEET_THRESHOLDS and passes(baseline, naive_target)
     and passes_on(holdout):
       return model_change_only               # just bump the model ID

  # ── Iterative repair ───────────────────────────────────
  original ← current editable files            # frozen, for diff sizing
  best ← naive_target ; best_files ← {} ; best_size ← 0
  width ← G.num_candidates ; widened ← false   # adaptive search width

  for i in 1 .. W.migration.max_iterations:
      clusters   ← cluster_failures(best.rows)  # group similar errors
      context    ← { clusters, failing & correct examples,
                     attempt history, editable + readonly files }
      candidates ← G.generate(context, escalated_width if widened else width)
      if candidates = ∅: break

      improved ← false
      for patch in candidates:
          size ← diff_size(patch, original)
          try:
              check_scope(patch)               # reject edits outside files.editable
              cand ← evaluate(M_target, apply(patch), tuning)
          except error:                        # patch broke the workflow
              log(failed) ; continue           # skip it — never abort the run

          better ← score(cand, O) > score(best, O)
          tie    ← score(cand, O) = score(best, O) and size < best_size
          if better or tie:                    # tie → the smaller edit wins
              best, best_files, best_size ← cand, patch.files, size
              improved ← true

      if O = MEET_THRESHOLDS and passes(baseline, best)
         and passes_on(holdout, best_files):
           commit(best_files) ; return pass    # validated on never-tuned data

      if improved:         widened ← false             # progress → cheap width
      else if not widened: widened ← true ; continue   # stall → widen once
      else:                break                        # stalled at full width

  # ── Resolve outcome ────────────────────────────────────
  if O = MAXIMIZE:                              # refine
      validate best_files on holdout (no-regression vs. current)
      suggest fresh thresholds from holdout metrics
      return pass if best beats naive_target else no_change
  else:                                         # migrate, thresholds unmet
      return partial if best improved over naive_target else blocked

Invariants the loop guarantees, regardless of what a generator proposes:

• Sandboxed trials — every candidate is applied via backup → run → restore; the working tree is written only on a committed pass.
• Crash isolation — a candidate that breaks the workflow (e.g. emits invalid YAML/JSON) is logged as a failed attempt and skipped; it can't abort the run.
• Minimal-change tie-breaker — on an exact score tie the smaller edit wins; against the no-op baseline (best_size = 0) a same-scoring patch is rejected, so the loop never makes a change that doesn't help.
• Stall-escalation — a stalled iteration widens the candidate pool once (to max(width × 3, 5)) before giving up — cheap when easy, broad when stuck.
• Holdout gate — nothing is committed until it clears a split it never tuned against.

Migration statuses

migrate exits non-zero on partial / blocked, so it gates CI naturally. A blocked migration still produces a full report and files an actionable issue.

Safety guarantees

These are enforced by the engine, not left to the patch generator:

• Edit-scope enforcement — any patch touching a file outside files.editable is rejected before it is applied.
• Sandboxed application — candidate edits are applied with originals backed up and always restored, so evaluation never leaves the repo dirty.
• Holdout gating — nothing is committed unless it passes thresholds on the holdout split.
• No auto-merge, no force-push — open-pr is a dry run by default; --create opens a PR/issue but never merges or pushes to the base branch.

Known limits

• The supported surface is the CLI plus composite Action in CI you control. There is no hosted bot; that is out of scope, not an unfinished 1.0 feature.
• Embedding migrations and vector-index rebuilds are out of scope; RAG migrations keep retrieval fixed.
• Agent tools should be fake or sandboxed, and provider lifecycle data is bundled/CI-refreshed rather than a live hosted catalog.
• Small evals can produce noisy passes. Use representative data, meaningful holdout, and multiple seeds for high-risk changes. See eval confidence.
• --generator fixture only repairs bundled examples. Customer workflows use --generator llm.

Read the full limits and ownership boundaries before broad rollout.

Cost & budgets

compare runs current and target workflows. migrate adds candidate evaluations and holdout; judge grading can add one provider call per output on every evaluation. Start with a representative sample, migration.max_iterations: 2–3, one candidate at a time, and a required holdout. Expand only after the report shows useful progress.

Estimate rows × models × candidates × iterations (plus judge/tool calls), set provider-side spend limits, and keep open-pr dry-run until the result is understood. See cost shapes and suggested starting budgets.

Upgrading

Read the changelog before changing release lines. Version 0.3 rejects legacy migration.allow_* fields: replace them with exact files.editable paths, run validate locally, preview one compare, and only then update CI. Follow the complete 0.3 upgrade guide.

Triggers & policy

The rest of the tool answers can we migrate; the policy layer answers when should we — the "Dependabot config" of the project. A trigger is only a candidate; whether it's worth it is decided by running your eval.

A .driftless/policy.yml configures per-trigger thresholds, candidate allow/deny globs (preview models denied by default), and an ignore snooze list. The plan command wires this together as a CI triage step.

CLI commands

Useful flags on migrate:

• --generator llm|none|fixture — repair strategy (LLM-backed by default; none is a dry analysis; fixture reproduces bundled-example patches).
• --to <model> — the target model to migrate to (otherwise the contract's candidates are used).
• --strict-label-audit — block when duplicate/near-duplicate inputs disagree on gold labels (warns by default).

Contract schema reference

Evaluation metrics

compare and migrate load the output JSONL your command writes, align it with gold labels, validate each record against the JSON schema, and compute:

By default they compare current prompt on current model vs current prompt on target. When the prompt was never source-optimized, that delta mixes prompt debt with model drift — see Measuring migration gains.

• Accuracy + macro precision / recall / F1 (per-class breakdown retained for failure clustering).
• Schema error rate — unparseable or schema-invalid records.
• Refusal rate — empty/null labels, a truthy refused field, or values listed in eval.refusal_values.
• Average latency — derived from run duration / record count.
• Total cost — only when the workflow emits a per-record cost_field. Token-based estimates are never fabricated.

Measuring migration gains

compare and migrate score your current prompt on the current model (baseline) and the same prompt on the target (naive_target). That mirrors flipping a model ID in prod without touching the prompt — the right default.

But when the prompt was never tuned to its ceiling on the source model, the delta conflates two effects:

• Prompt debt — under-optimization that would improve on either model.
• Model-induced drift — quality lost because the target behaves differently.

A headline "+0.07 F1 after migration" can be mostly debt, not repair. The refine path (dataset change, model pinned) avoids this; model migration needs a control.

2×2 control

Optimize on the source model first, then switch. Example from the testbed (macro-F1, 290 labels, real API calls, gpt-3.5-turbo → gpt-4o-mini):

• P0 → A = prompt debt on the source (not migration).
• A → B (−0.072) = true model-induced drift from a strong baseline.
• B → D (+0.066) = gain from re-tuning after the switch.

Full write-up and repro steps: Measuring migration gains honestly.

Run viewer

After migrate or refine, inspect the optimization trajectory in a local web UI — iteration metrics chart, failure-cluster trends, and a full attempt log (rationales, scores, diff sizes, accept/reject).

copydriftless view                    # opens http://localhost:8777/runs.html
driftless view -w support_classifier

The viewer reads .driftless/migrations/<workflow>.json from the current project. You can also load any result JSON via file picker or drag-and-drop. Static demo: runs.html → Load saved passing fixture. Reproduce that passing path locally with migrate --generator fixture on a bundled example. The --generator none quickstart still ends BLOCKED.

Repair & custom generators

The engine defines a PatchGenerator protocol; the repair strategy is swappable. Three ship out of the box:

• LLMPatchGenerator (default) — asks an LLM to rewrite the editable files to fix the observed failure clusters. Provider-neutral, requests strict JSON, and varies temperature across candidates.
• NoOpPatchGenerator (--generator none) — proposes nothing; the loop becomes a dry analysis, useful offline and for CI gating.
• FixturePatchGenerator (--generator fixture) — applies the known-good patch for a bundled example so a passing repair is reproducible without provider keys.

You can customize repair via the contract's repair: block — append domain guidance, fully replace the system prompt, or supply a user_template with {{placeholder}} substitution (placeholders include failure_clusters, failing_examples, editable_files, metrics, and target_model).

A generator only ever proposes — the engine owns acceptance. See The migration loop for the full algorithm (sandboxing, crash isolation, the minimal-change tie-breaker, stall-escalation, and holdout gating).

GitHub Action

A composite GitHub Action wraps the CLI so scans and migrations can run in CI. After validating locally, run driftless init-ci, review the generated files (and any inferred setup step), and commit only the automation you need. Override setup with --setup-command when inference does not fit. Refinement remains manually dispatched unless --refine-on-push is explicit.

copy# .github/workflows/driftless-plan-preview.yml
name: Driftless plan preview
on:
  schedule: [{ cron: "0 9 * * 1" }]
  workflow_dispatch: {}

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: driftless-dev/driftless@v0.3.4
        with:
          command: plan

This is a policy plan preview, not the dependency-only scan workflow. A scheduled plan gates CI when a deprecated model needs attention; a manually-triggered migrate opens a PR (or an issue when blocked) with the evidence attached.