Support changed the labeling policy — model stays put

The use case

Same support classifier as post 1: every
ticket gets a gold category so you can measure quality offline. For a long time,
tickets like "Please reverse the payment on my latest invoice" are labeled
billing — the team treated a charge correction as a billing adjustment, not
a refund.

Then support ops updates the labeling policy. After a product decision, those
charge-reversal tickets should be refund, not billing. An engineer runs a
script (or an annotator edits the JSONL), commits ~25 relabeled rows, and merges.
Nobody changes the model ID in config. Production still calls the same model with
the same prompt.

On the next eval run, accuracy drops. Dashboards look like a model regression.
Someone proposes swapping to a "smarter" model. That is the wrong diagnosis: the
oracle moved. The prompt still teaches the old definition of billing vs refund,
so the pinned model is now wrong for the new definition of "correct."

This is dataset drift, not model drift. The job is to re-tune the prompt against
the updated labels while keeping the model fixed — not to chase a new model ID.

What driftless does here: pin the model, re-optimize allowed prompt files
against the new gold labels (refine), validate on holdout, and open a PR.

Again we use
support-classifier-svc
— same 290-ticket classifier, same driftless.yml, different trigger.

Same repair loop, different variable

If you bump the model when only labels changed, you're debugging the wrong knob.

Use the trigger to choose the tool:

The policy change (concrete tickets)

The dataset builder seeds 25 charge-reversal tickets. They read like billing
but, after a policy meeting, should be refund.

Example rows from evals/tickets.inputs.jsonl:

{"id": "t002", "text": "I need you to reverse the credit card payment because it was charged twice."}
{"id": "t021", "text": "Please reverse the payment on my latest invoice."}

Before policy: gold label billing (adjustment on a charge).
After policy: gold label refund (customer wants money back on a charge
they dispute).

The testbed ships a one-command event script — no hand-editing JSONL:

python evals/_apply_refund_policy.py
# policy update: re-labeled 25 charge-reversal ticket(s) -> 'refund'

That script uses the same detector as the simulator
(support_classifier.llm_client._is_charge_reversal) so offline runs stay
reproducible. On main today the policy may already be applied (running the
script prints 0 changes); for a fresh demo, start from a commit before the
policy or restore labels from git history.

Then:

export SUPPORT_CLASSIFIER_SIMULATE=1
driftless refine -w support_classifier --strict-label-audit
driftless open-pr -w support_classifier --create

No --to. The model in config/llm.yml never changes.

The resulting PR should be boring in the best way: label/input diff in one
commit or branch, prompt diff in the Driftless PR, model config untouched, and a
report that says the repaired prompt meets the current eval policy.

What the repair actually edits

The simulator is calibrated so the old prompt scores well on the old
labels and poorly after relabeling — until category definitions catch up.

A successful real-model repair (documented in the testbed README, scenario 3)
rewrote prompts/system.md from:

- billing: questions about invoices, charges, payments, or subscriptions
- refund: the customer wants their money returned

to something like:

- billing: ... including requests to reverse or correct erroneous charges
- refund: ... charged correctly but dissatisfied or no longer wish to pay

Observed on live gpt-4o-mini (repair generator: gpt-4o, 290 tickets):

Why so low before? Real models label charge reversals refund ~92% of the time
from priors alone — but this product's policy says many reversals are billing.
The prompt never stated the rule; the model wasn't "broken," the spec was
implicit.

Counterintuitive gold labels are common in production (password-reset →
security, not account). refine is built to discover explicit rules from
failure clusters, not just add a few-shot example.

Scenario 4: feedback batch (messier, more realistic)

Pure relabels are rare. The testbed also ships
evals/_apply_feedback_batch.py:

• +22 new tickets — subscription cancellation / lifecycle requests missing
  from v1 of the eval set
• 2 relabels — general subscription inquiries billing → account

On real gpt-4o-mini, account recall on the new tickets goes from 0.11
(16/18 predicted billing) to 1.00 after refine adds a lifecycle rule.
Offline simulator: accuracy 0.936 → 1.000.

Same command path: apply script → refine → PR. Exercises poll's meaningful-
change detection if labels live outside git.

CI: path filter on the file that actually changed

In-repo eval data → git is the change detector. The testbed workflow
refine-on-label-change.yml:

on:
  push:
    branches: [main]
    paths:
      - "evals/tickets.labels.jsonl"
      - "evals/tickets.inputs.jsonl"

Notice prompts/ is not in paths — otherwise the refine PR would
re-trigger itself.

Job steps (abbreviated):

1. driftless audit-labels -w support_classifier --fail
2. driftless refine -w support_classifier --strict-label-audit with
   SUPPORT_CLASSIFIER_SIMULATE=1 (harness offline; repair generator still
   uses OPENAI_API_KEY when set)
3. driftless open-pr -w support_classifier --create
4. Upload .driftless/reports/ to the Actions summary + artifacts

Try it: commit a label change locally, push, or dispatch the workflow from
Actions.

Preflight: contradictory gold labels

Before spending tokens:

driftless audit-labels -w support_classifier --fail

Near-duplicate ticket text with different gold categories caps achievable F1.
The testbed runs audit on every PR touching evals/tickets.*.jsonl
(audit-labels.yml).

More: post 5 outline.

This is the boundary between data cleanup and prompt repair:

Simulator vs real API (when to use which)

Scenario 2 (charge-reversal relabel) is reproducible offline. Scenario 3
(counterintuitive billing/refund policy on unchanged labels) only bites on
real models — that's why the testbed has both Refine on label change
(simulator) and Real-model refine (weekly/manual, real API).

Next steps

• Post 1: model deprecation → migrate
• Post 3: weekly plan across both workflows → CI post
• Clone the testbed, run _apply_refund_policy.py, then refine