Prompt repair that doesn’t wait for someone to remember

The problem and the outcome

Your prompts, evaluation data, and Python test harness already live in Git. You
can test a model change or update a prompt after labels change, but only when
someone remembers to run the commands.

That manual loop breaks down quickly. A provider announces a model retirement.
Someone changes the model ID under deadline pressure. Annotators update labels,
but continuous integration (CI) never checks whether the prompt still matches
them. Cheaper models go unnoticed because no one owns weekly review.

The desired outcome looks like Dependabot for prompts: automation watches model
lifecycle and evaluation data, runs your real tests, and opens a pull request
(PR) or issue with evidence. A human still decides what ships.

Audit the labels first. Gold labels are the expected answers in your evaluation
set, and automation is unsafe when those answers contradict each other. Follow
post 5 before automating repair.

Mental model

Driftless separates discovery, decisions, and action:

• A scan is a read-only search for likely LLM workflow files, model IDs, and
  lifecycle risks. It does not create driftless.yml or edit your repository.
• A trigger is a condition that deserves review, such as a retired model, a
  changed dataset, or a cheaper active model.
• A policy is the YAML file that turns those conditions into rules: how
  early to warn, what savings matter, and whether the result should be a PR,
  issue, or dry run.
• A plan is the decision table produced by applying that policy to detected
  triggers and evaluation results. It routes work; it is not approval to ship.
• A scheduled workflow is a GitHub Actions workflow started by a clock, such
  as every Monday, rather than by a commit. GitHub Actions is GitHub's service
  for running repository automation; a workflow is a YAML file containing one
  or more jobs and their command steps.

driftless init-ci generates standard GitHub Actions workflows for review.
The external testbed later in this post uses hand-authored workflows instead.
Those testbed files include custom path filters, manual inputs, simulator setup,
and staged rollout behavior that a generic generator cannot infer.

Prerequisites

• Basic Python and Git command-line use
• A repository with a repeatable evaluation harness
• A Driftless workflow contract in driftless.yml
• Trustworthy labels before any automated repair
• Provider and GitHub credentials only when acting on a plan

Walkthrough 1: verify the bundled fixture

The bundled demo has four deterministic rows and needs no provider key. Install
Driftless, copy the example, validate its contract, compare its current model
with gpt-4o-mini, then generate a starter policy and CI workflows:

pip install driftless
driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini
driftless init-policy
driftless init-ci

The four-row comparison reports that the candidate fails the configured
thresholds. compare can still exit with code 0: read its threshold table and
the “Naive target does not pass” message instead of treating process success as
quality success. BLOCKED is a status used by migrate or refine when a
repair run cannot produce a shippable result; it is not the status of this
compare command.

Review the generated .driftless/policy.yml and
.github/workflows/driftless-*.yml files before committing them. This fixture
verifies command wiring, not provider behavior or a successful repair. Acting
on plans and LLM-backed repair require credentials; --create adds GitHub side
effects.

EXAMPLE_SUCCESS_PR.md distinguishes public
testbed PR #4 from the different bundled four-row saved success fixture. Both
show the open-pr evidence shape; see the
repair reproduction boundary.

Open the public draft PR and inspect the generated report and diff.

Walkthrough 2: discover an existing application

Run scan before configuration so you can see what Driftless can detect without
changing files. Then create a draft workflow contract, complete its TODOs, merge
it into the root driftless.yml, and validate it:

driftless scan .
driftless configure support_classifier .
# complete the generated TODOs and merge the draft into root driftless.yml
driftless validate -w support_classifier

The scan should list probable LLM files and known model lifecycle risks.
Detection is best-effort: models hidden behind environment variables or
gateways may need manual configuration. Validation should pass only after the
contract correctly names your harness, model, inputs, labels, and thresholds.

For a new repository, generate policy and workflow files after local behavior is
understood:

driftless init-policy    # writes .driftless/policy.yml
driftless init-ci        # writes .github/workflows/driftless-*.yml

These are generated starting points. The external
support-classifier-svc
testbed instead dogfoods hand-written workflows pinned to driftless==0.3.2.
Use its workflow directory
when you need examples of testbed-specific path filters, manual-dispatch inputs,
simulator setup, or staged rollout.

Walkthrough 3: preview weekly work in the 290-row testbed

The external testbed is a separate clone with two workflows in one
driftless.yml:

Its committed .driftless/policy.yml says:

deprecation:
  enabled: true
  warn_before_days: 90
  action: pr

cost:
  enabled: true
  min_savings_pct: 0.20
  max_quality_drop: 0.01

data_change:
  enabled: true
  min_changed_rows: 5

Both workflows still use deprecated gpt-3.5-turbo, so plan always has work.
The hand-authored plan-preview.yml
runs every Monday. Its simulator makes the run free and deterministic:

env:
  SUPPORT_CLASSIFIER_SIMULATE: "1"

steps:
  - run: driftless plan || test $? -eq 1
    # exit 1 = triggers found (expected here)

To reproduce that preview locally, enter the separate clone, enable its
simulator, and build the plan:

cd support-classifier-svc
export SUPPORT_CLASSIFIER_SIMULATE=1
driftless plan

Actual output from July 2026:

┃ Workflow          ┃ Trigger     ┃ Migrate                    ┃ Naive     ┃ Decision      ┃
│ support_classifier│ deprecation │ gpt-3.5-turbo -> gpt-4o-mini │ regresses │ ISSUE (critical) │
│ quick_triage      │ deprecation │ gpt-3.5-turbo -> gpt-4o-mini │ regresses │ ISSUE (critical) │

Why:
  gpt-3.5-turbo retired 277d ago; candidate gpt-4o-mini not shippable as-is
  (status=blocked) -> open issue

2 workflow(s) need action across 1 model move(s):
  gpt-3.5-turbo -> gpt-4o-mini (deprecation): support_classifier, quick_triage

The result groups one model move across two workflows instead of opening two
blind migration PRs. Reports land under .driftless/reports/. The decision is
ISSUE, not PR, because the naive comparison fails thresholds with 100%
schema errors in the simulator; post 1
explains why migration and repair are required.

Interpret decisions this way:

Walkthrough 4: add action carefully

The hand-authored plan-act.yml
uses manual dispatch, meaning a person starts it from GitHub's Actions page
and can provide inputs. First preview the Git and GitHub operations:

driftless plan --act

After the preview matches team expectations, allow creation:

driftless plan --act --create

--act runs migration or refinement work but remains a dry run by default.
--create performs Git and GitHub operations and can open PRs or issues. It
needs OPENAI_API_KEY for LLM-backed repair and GH_TOKEN for GitHub.
Scheduled runs stay dry-run; only a manual dispatch with create=true opens
PRs. Keep the first runs as calibration while the team learns which conditions
deserve automation.

Walkthrough 5: react to label and model changes

An event-driven workflow runs because a repository event occurred. In the
testbed, refine-on-label-change.yml
runs after a push changes the label file:

push to evals/tickets.labels.jsonl
  → audit-labels --fail
  → refine --strict-label-audit  (simulator harness)
  → open-pr --create

The audit-first ordering prevents repair from optimizing against contradictory
labels. Post 2 covers the
charge-reversal policy fixture: evals/_apply_refund_policy.py changes 25
tickets.

For a real model move, the hand-authored
migrate-on-model-change.yml
is started through Actions → Migrate model → Run workflow:

Its order is audit-labels → real compare → migrate --strict-label-audit → open-pr --create. It requires OPENAI_API_KEY and
allows 120 minutes for LLM repair over 290 tickets. The resulting PR includes a
scorecard, changes to prompts/system.md and config/llm.yml, a holdout result,
and an attempt log.

Finally, audit-labels.yml
runs on PRs that touch evals/tickets.*.jsonl and blocks merge when similar
inputs disagree on gold labels.

Failure and safety behavior

• scan is read-only and best-effort.
• plan exits non-zero when it finds work, which is expected in the preview
  workflow.
• A failed naive comparison becomes an issue rather than an unsafe PR.
• Scheduled plan --act remains dry-run unless creation is explicitly enabled.
• --create has external side effects and requires credentials.
• The four-row fixture is a smoke test; it is not evidence about provider
  behavior. The 290-row testbed is a separate clone with different fixtures.

Suggested rollout order

1. Run audit-labels on evaluation file paths
   (post 5).
2. Add refine-on-label-change only after the audit is clean
   (post 2).
3. Run plan-preview weekly with the simulator for visibility without keys.
4. Use migrate-on-model-change by manual dispatch when you accept token cost.
5. Enable plan-act --create only after policy thresholds match team risk.

The testbed workflow summary is:

Next steps

• Post 1: compare and migrate
• Post 2: refine after a label change
• Post 4: cost trigger — this needs an
  active baseline model; testbed deprecation rows dominate until migration
• Product workflows: Driftless .github/workflows/