Practical guides for LLM workflows that move.

You run a support product that routes every inbound ticket through an LLM classifier before a human ever sees it. The model returns a small JSON object like {"category": "billing"}. Downstream code parses that JSON…

Same support classifier as post 1: every ticket gets a gold category so you can measure quality offline. For a long time, tickets like "Please reverse the payment on my latest invoice" are labeled billing — the team…

Your LLM app already looks "done" from a git perspective: prompts live in the repo, gold labels live in JSONL, and a harness command can score quality offline. You can migrate a model or refine after a label change…

Your classifier (or similar LLM workflow) runs on a frontier model — say gpt-4o — because that is what cleared quality when you launched. Volume grew. Finance now wants 20%+ inference savings. A cheaper model like…

You trust your offline eval. Macro-F1 on a few hundred labeled tickets is how you decide whether a model migration or prompt refine is safe. After a labeling policy change (or a messy annotation pass), you run…

Not every LLM workflow is a classifier with gold categories. You might summarize support tickets, rewrite answers for tone, or grade free-form responses for faithfulness. There is no single label_field to compute…

You ship a retrieval-augmented QA feature: user asks a question, the app retrieves chunks from a knowledge base, and an LLM writes an answer that should stay grounded in those chunks and cite them. Quality is…

You run a support-style agent that must use tools — look up an order, check a policy, create a ticket — before it answers the user. Success is not only a polite final message. The agent must call the right tools in a…