any-llm walks · 23 Oct 2026 · Bristlenose 0.42.0 · any-llm-sdk 1.27.0

Which models through any-llm are good enough, cheap enough and fast enough?

Thirteen runs on the same three oral-history interviews, each scored against Claude Sonnet 4.6. Agreement is how much of the transcript a model's quotes cover in common with Sonnet's. Sonnet run twice agrees with itself 90% of the time, so no model can be told apart from Sonnet above that line.

Nothing reached Sonnet's own 90%.The best of the rest landed at 80–82%. Sonnet 5.5 is one of them, at the same price as 4.6.

Cheap and close means a known flaw.GPT-5.4 mini and DeepSeek V4 Pro reach 82% at about a fifth of Sonnet's price; DeepSeek Flash 77% at about a twelfth. Each one invents screens, over-splits themes or over-extracts.

The route changes the model.DeepSeek Flash native: 77% in 2m 20s. Through OpenRouter it thinks by default: 63% in 7m 19s. Qwen with thinking on never finished.

No model made quotes up.96–100% of every model's quotes are verbatim in the transcript. The failures are coverage and structure, not invention.

Read with care

One run per model (Sonnet 4.6 twice, for the noise floor). Gaps of a few points are indicative, not settled.
Three FOSSDA interviews (s1, s6, s10): public oral history, not usability testing. One IKEA usability session is shown separately below.
Agreement measures sameness to Sonnet 4.6 run A, not correctness.
Prices are list price × recorded tokens, not invoices. Cancelled calls record no tokens and aren't counted.
Runs span several commits on 23 Oct 2026; the Qwen "thinking off" row is after 8b16e797f.

Agreement with Sonnet 4.6 against list price, FOSSDA interviews

Hover or tab to a point for its time, quote count and themes. The left column holds runs with no per-run price recorded; the Mistral walk's whole bill, both models and all 72 requests, was €0.13. Sentiment agreement isn't charted, because Sonnet agrees with itself on sentiment only 64% of the time.

The same comparison on one usability session (IKEA, 18 minutes)

The noise floor here is 85%. This session has real screens: Sonnet 4.6 found 10, and the shipped report, built with screen text from the video, has 8. Models that collapse the flow to 3–5 screens lose the journey. Mistral Large through OpenRouter failed at 48 s, rate-limited at the host's shared pool.

Every FOSSDA run, by agreement

Model	Route	Agreement	List price	Wall time	Quotes	Themes	Screens	Verbatim

Screens should be 0: these interviews have none. Wall time is on a log scale from 20 seconds to 40 minutes. Verbatim is the share of quotes whose words are at least 90% in the transcript.

How they fail

Invented structure

On interviews with no product screens, GPT-5.4 mini made 7, Llama 3.3 70B and DeepSeek Flash 6 each, DeepSeek V4 Pro 1. A quick tell of a model that fills a schema slot whether or not the data supports it.

Too many or too few themes

DeepSeek V4 Pro split the material into 23 themes, nine with 2 quotes; Sonnet made 12–15. GPT-5.4 mini made 3.

Thinking by default

Hybrid models think unless told not to, and the thinking is billed as output. A short probe on Qwen 3.8 Flash took 41.6 s and 1,839 output tokens; with thinking off, 2.1 s and 91. Its whole run went from killed after 38 minutes to 2m 56s. GPT-5 mini and Gemini 2.5 Pro refuse to switch it off.

A dropped field

Mistral Small left out a required field on one quote, and the whole session's quotes were discarded: it lost s1 in 3 of 3 attempts. Since 5939d6c89 the valid quotes are kept and the dropped ones are named.

The host, not the model

Free models failed at the host: a 404, a 429, and a Nemotron run that gave up after 29m 47s. A shared-pool rate limit ended Mistral Large through OpenRouter at 48 s. Llama 3.3 70B took 13 minutes on an 18-minute session.

Confidence without fit

Coverage hides calibration. In AutoCode with a usability codebook that doesn't fit oral history, Sonnet gave all 70 quotes 0.10–0.20 and excluded them. Mistral Large gave 0.3–0.7 to all 35, accepting 4.