Generated 2026-09-18 01:27:04 PDT from benchmarkings/task_classifier/results.json.
Before KISS Sorcar starts a task it classifies the prompt twice over: is this simple (neither
software development nor Internet search, so the lite system prompt suffices) and is this development
(it will create or edit files, so the run gets its own git worktree). Until now one non-agentic call on the
run's own LLM produced that verdict. This benchmark compares it with a classifier that asks OpenRouter's
~typesafe/jev-latest decisions model a single typed choice question — which of
five kinds of task is this: development, git-only, internet, simple, or an ambiguous follow-up — and maps
the answer to the same two booleans. A chosen kind with probability below 0.6 is treated as ambiguous
(the conservative verdict: full prompt, worktree kept).
The prompts are 415 tasks: 361 real, de-duplicated
user prompts taken from a KISS Sorcar history database (paths and personal data removed) and
54 synthetic prompts covering edge cases the real sample
is thin on (git-only chores, weather and price look-ups, bare fragments such as fix it). Each
was labelled by hand against the classifier's own definitions; 344 labels are clear-cut and
71 are marked uncertain because a careful reader could defend either verdict
(for example "authenticate gmail", or a follow-up question whose context is missing). Category counts:
development 186, simple 128, git_only 40, internet 44, ambiguous 17.
Read the numbers with this in mind. The labels and the Jev question
wording were written by the same hand, from the same definitions, so the comparison favours Jev in the
way any benchmark favours the system whose criteria it was written against. The LLM prompt is the
production prompt, unchanged. Two mitigations: the Jev wording and the probability floor were chosen on
one half of the prompts (201 tasks, split = dev) and checked on the other; recomputed
from the raw answers in results.json, Jev's accuracy without the floor is
87.6% on the tuning half and 89.7% on the held-out half,
and with the floor 85.1% and 91.1%: the held-out half
scores at least as well as the tuning half either way, so the choices did not fit the tuning half in
particular. The accuracy on the clear-cut labels is also reported separately.
| classifier | accuracy | accuracy, clear-cut labels | is_simple | is_development | lite-prompt errors | no-worktree errors | no verdict | cost / task | total cost | median latency | p90 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| jev-decisions | 88.2% | 94.2% | 89.6% | 90.8% | 11 | 5 | 4 | $0.000025 | $0.0102 | 0.22 s | 0.61 s |
| claude-fable-5-1 | 80.0% | 89.2% | 80.7% | 89.9% | 4 | 1 | 7 | $0.004953 | $2.0557 | 3.46 s | 4.77 s |
| claude-haiku-4-5 | 59.8% | 67.7% | 65.8% | 90.6% | 8 | 27 | 0 | $0.000412 | $0.1710 | 0.58 s | 0.70 s |
| gpt-5.6-sol | 69.4% | 76.7% | 70.8% | 92.3% | 7 | 6 | 0 | $0.002035 | $0.8446 | 2.33 s | 4.46 s |
| gemini-3.5-flash | 77.6% | 86.3% | 79.3% | 90.4% | 6 | 7 | 0 | $0.003178 | $1.3187 | 3.67 s | 5.52 s |
Lite-prompt errors: tasks that are not simple but were judged simple, so they would run on the lite prompt. No-worktree errors: development tasks judged non-development, so they would run without worktree isolation. These are the two mistakes that can hurt a run; the opposite mistakes only cost a full prompt or an unneeded worktree. No verdict means the classifier failed and the run kept its defaults. Recorded causes:
jev-decisions: 4 × Error: Decisions request to https://openrouter.ai/api/alpha/decisions failed: HTTPSConnectionPool(host='openrouter.ai', port=443): Read timed outclaude-fable-5-1: 7 × Task classification attempt failed: ModelRefusalError: KISS Error: Model claude-fable-5-1 refused the request for safety reasons (stop_reason="refusal", empty response)Per task, Jev costs $0.000025 against $0.004953 for claude-fable-5-1, a factor of about 202; its median latency is 0.22 s against 3.46 s, about 15× faster. Because a classification happens before every task, the latency is felt directly as time-to-first-action.
The LLM classifiers are conservative by instruction: the prompt tells them to answer
{"is_simple": false, "is_development": true} whenever in doubt, and they take that
liberally. claude-haiku-4-5 and gpt-5.6-sol call almost every git-only task development (1/40 correct and 2/40 correct), and every LLM labels fewer than half of the simple tasks as simple.
That is safe — a worktree and the full prompt never break a task — but it forfeits most of
the speed-up the classifier exists to provide. Jev, asked a contrastive five-way question, separates
git-only chores and code-reading questions from development far more reliably, at the price of more of
the harmful kind of error: 11 non-simple tasks on the lite prompt and
5 development tasks without a worktree. They split two ways:
open research questions that Jev reads as plain questions ("what is the simplest and most powerful memory
system for an AI agent?", "how do I complete a drive cycle on a 2017 BMW?"), and behaviour requests or
pasted error logs that it reads as chores or git operations ("after run_parallel finishes, you must show
the results", "I am getting the following error when I run ./rsorcar. fix it." followed by a git push
log).
| pair | same verdict |
|---|---|
| jev-decisions vs claude-fable-5-1 | 81.2% |
| jev-decisions vs claude-haiku-4-5 | 59.0% |
| jev-decisions vs gpt-5.6-sol | 69.2% |
| jev-decisions vs gemini-3.5-flash | 77.6% |
| claude-fable-5-1 vs claude-haiku-4-5 | 67.0% |
| claude-fable-5-1 vs gpt-5.6-sol | 78.6% |
| claude-fable-5-1 vs gemini-3.5-flash | 82.4% |
| claude-haiku-4-5 vs gpt-5.6-sol | 81.9% |
| claude-haiku-4-5 vs gemini-3.5-flash | 70.1% |
| gpt-5.6-sol vs gemini-3.5-flash | 77.6% |
The LLMs agree with each other about as often as they agree with Jev, which is a reminder that the task is genuinely ambiguous at the margins: no two classifiers here agree on more than 82% of prompts.
why didn't you find https://ntfy.sh/kiss-31e7ee3ddff0e4f3ef6780754e4159ed?how is the agent classifier implemented?does run_agent tool calls the agent classifier?what is teh signature of the `run_agent` method?How many lines of code are there in the project?Once the GitHub Education teacher application is approved and Berkeley-CS-264 is upgraded to the Team plan, re-run `gh teacher init Berkeley-CS-264 --yes`, set …can you find receipts from Plus Docs or that company in my gmail?authenticate whatsapphow jev can be effectively used with a ReAct agent harness for LLM?Review the git diff of src/kiss/core/models/MODEL_INFO.json and README.md and commit the model catalog update.can you find all channels on nfty.sh whose name begin with kiss-?what jev model and how does it work and how can I use it?can you create a git lassroom for me for the course Cs 264 I am teaching at UC Berkeley so that students can submit their assignments?why the file paths in the result of the last task are clickable?in ./src/kiss/agents/third_party_agents/README.md, can you say how to run a always on gateway from the KISS Sorcar chat without asking it to run a kiss-* comman…Register a device-flow-enabled GitHub OAuth app and a Twitch public app, set KISS_GITHUB_CLIENT_ID, and run the new Connect sign-ins end to end against the real…classify_task now asks Jev first whenever an OPENROUTER_API_KEY is
configured and the classify_with_decisions config key (default on) allows it, through the same
decide tool agents use. Without a key, with the key switched off, or when the call fails
for any reason (including the ~1% of requests that hung in this run, which production cuts off after
5 seconds), the LLM classifier runs exactly as before.cc/*, codex/*) that the LLM classifier must skip.uv run python benchmarkings/task_classifier/run_benchmark.py --llm-model
claude-fable-5-1 … then uv run python benchmarkings/task_classifier/make_report.py.