Why KISS Sorcar trails the HarnessTax agents, and how to close the gap

A post-mortem of the 1,260-attempt replication of the HarnessTax study (7 models × 2 benchmarks × 30 tasks × 3 repetitions), built from the trial trajectories, the official SWE-bench and Harbor verifier logs, and the 2.25 million events the KISS daemon recorded in ~/.kiss-harnesstax/sorcar.db. Dated 2026-09-20.

1. Which KISS Sorcar agent ran the comparison

Every attempt was a task submitted to a private KISS daemon (KISS_HOME=~/.kiss-harnesstax) and executed by the standard Sorcar agent (src/kiss/agents/sorcar/sorcar_agent.py, wrapped by relentless_agent.py and built on KISSAgent). The Sorcar agent was loaded with a Sorcar Extension Agent (SEA) generated per trial from benchmarkings/harnesstax/sea_core.py. The SEA replaces Sorcar's normal tool set and system prompt with a deliberately minimal harness modelled on the blog's Pi harness:

• Four tools plus finish: bash (fresh shell per call, 180 s default / 1,800 s maximum timeout, output truncated to 30,000 characters), read (numbered lines), write, edit (exact single-occurrence replacement). Every call is a docker exec in the task container.
• A ~1,000-character system prompt ("explore, reproduce, make focused changes, verify, do not stop early, never modify existing tests").
• A 100-turn cap enforced through the SEA's llm_call_hook, a $15 budget per attempt, and the task's own wall-clock limit (Harbor's [agent] timeout_sec minus 45 s on TB2).
• No web search, no memory, no worktrees, no sub-agents, no decide(); the same model at the provider's default reasoning setting (adaptive thinking for Fable 5 and Opus 4.8, a 10,000-token thinking budget for Sonnet 4.6 and Haiku 4.5, reasoning: high for the GPT-5.6 models).

What the Sorcar wrapper still contributes on top of the SEA matters for the analysis below: it appends its own "MOST IMPORTANT INSTRUCTIONS" block (~1,100 characters about finish(success=False, is_continue=True, summary_in_html=…), the work directory and the process id) to the system prompt, appends a usage trailer (Steps: N/10000, Context: …, Budget: $…/$15.00) to every tool result, requires an HTML summary and a suggested next task in finish, and runs the relentless session loop that restarts a fresh session when a session ends without success=True. The 10,000 in the trailer is Sorcar's step limit, not the SEA's 100-turn cap; the model never saw the real cap or the remaining wall-clock time.

2. How far behind is it, really

Averaged over the seven models, KISS Sorcar is 0.5 points behind Pi on SWE-bench Lite and 2.1 points behind on Terminal-Bench 2, ahead of Codex and Claude Code on Terminal-Bench 2 (67.9% vs 65.7% and 65.1%), and behind them on SWE-bench Lite (71.4% vs 73.7% and 73.4%). Per model it is within a few points of Pi on both benchmarks; on SWE-bench Lite it is cheaper than Pi with Fable, Opus, Sonnet and Kimi and 20–40% dearer with Haiku, Sol and Luna. The visible gaps are against Codex and Claude Code with the two strongest Claude models on SWE-bench Lite (Fable 5: 93.3% vs 96.7% / 97.8%; Opus 4.8: 81.1% vs 88.9% / 86.7%) and against Pi on Terminal-Bench 2 with Opus 4.8 (63.3% vs 72.2%), Sol (−4.4) and Luna (−3.4). Each of those gaps is between 3 and 8 attempts out of 90.

SWE-bench Lite

Terminal-Bench 2

3. Where the lost attempts went

Each failed attempt was classified from its trajectory and verifier log. On SWE-bench Lite, 175 of the 180 failures are the agent calling finish(success=True) with a patch the hidden tests reject; only 5 hit the turn cap and none timed out. On Terminal-Bench 2 the picture is different: 125 wrong claims, but also 66 attempts killed by the task's wall-clock limit, 7 turn caps and 4 explicit give-ups.

3.1 The tasks that make the difference

Summing solved attempts over the seven models (21 per task) and subtracting Pi's count shows that the deficit is not spread evenly; it sits in a handful of tasks. The gains are concentrated too: KISS solves scikit-learn-13241 21/21 against Pi's 12/21 and feal-linear-cryptanalysis 14/21 against 11/21.

3.2 Root cause A — evaluation noise, not the agent (SWE-bench Lite, ≈7 attempts)

psf__requests-1963 accounts for the single largest gap: 9/21 for KISS against 20/21 for Pi. The official reports tell a different story than the headline number. In 7 of the 12 failed attempts every FAIL_TO_PASS test passed and the attempt was rejected only because a PASS_TO_PASS test that calls the live httpbin.org service (test_cookie_sent_on_redirect, test_unicode_multipart_post, test_BASICAUTH_TUPLE_HTTP_200_OK_GET, …) raised a JSONDecodeError on a non-JSON response. The Fable 5 attempt that "failed" contains the upstream fix verbatim:

+            # Update the request that is used for the next redirect
+            req = prepared_request
+
             resp = self.send(

With ~15 evaluations running concurrently from one IP, httpbin throttled some of them. Netting these out puts KISS on SWE-bench Lite at 72.5% against Pi's 71.9%. Over all SWE failures, 157 were genuine FAIL_TO_PASS misses, 12 were FAIL_TO_PASS plus a regression, and 9 were PASS_TO_PASS-only failures (7 of them this flake).

3.3 Root cause B — the wall clock on compute-heavy Terminal-Bench tasks (≈20 attempts)

The TB2 deficit is dominated by five tasks: compile-compcert (−6 vs Pi), gcode-to-text (−5), path-tracing (−4), make-doom-for-mips (−3) and torch-tensor-parallelism (−2). Except for the last, the failures are timeouts, and they have two distinct shapes.

Compute-bound. In compile-compcert, train-fasttext and path-tracing the tool time equals the wall time: the agent spends the whole budget waiting on opam install coq (4–8 minutes each), fastText training runs (2–14 minutes each) or a renderer. Three things compound here:

1. Host contention. The sweep ran a mean of 15 agent tasks at once (peak 26) on a single 32-core VM, each with its own container and its own make -j. The blog's harnesses ran one task per environment. A Coq build that takes 4 minutes on an idle machine took 5–8 here, and CompCert's own build never started before the 2,400 s limit.
2. The agent's own timeouts killed its own jobs. Across TB2, 129 bash calls were killed by the timeout the agent had passed after running for ≥ 2 minutes (8.8 hours of compute in total), and in 85 cases the next call re-ran essentially the same command from scratch. 22 of the 66 timed-out attempts contain such a kill. Nothing in the harness offered a way to run a job in the background and poll it.
3. No deadline awareness. The model saw Steps: 12/10000 in every tool result and never the remaining time. In train-fasttext Fable 5 had a model meeting both the size and its own accuracy target after 10 turns, then spent 45 more minutes on hyper-parameter sweeps and was killed mid-experiment; the verifier measured the last model it had left behind.

Latency-bound. gcode-to-text and make-doom-for-mips have a 900 s budget. Here the tool time is 1–4 minutes and the rest is the model: Opus 4.8 took 38–56 turns at 15–21 s each on G-code interpretation, Fable 5 24 turns at 34 s each with ~52,000 prompt tokens and ~3,200 output tokens per call. Event timestamps in sorcar.db rule out API queueing: median time-to-first-token was 0.6–5 s for every model, and of the 89 model calls that took longer than 2 minutes, 88 were streaming thinking tokens the whole time (100,000–400,000 characters of reasoning in a single call on path-tracing-reverse, feal-linear-cryptanalysis and adaptive-rejection-sampler). These are reasoning marathons, and every harness in the blog runs the same models at the same effort. The difference is that a 900 s task leaves no slack for them.

3.4 Root cause C — patches that pass the agent's tests and fail the hidden ones (model-limited)

The largest failure class by count, 175 SWE and 125 TB2 attempts, is the agent declaring success on a wrong solution. It is not a failure to test: every SWE attempt ran a test command, and in 86–100% of the wrong claims the last edit was followed by a test run. Compared with the gold patches, 89% of the wrong patches touch the right file and 73% touch only the gold files. The agent finds the place and implements a different reading of the issue than the one the hidden test encodes. The tasks are the usual suspects: astropy__astropy-7746 (19 wrong claims; Pi 2/21, Codex 5/21), sympy__sympy-11400 (17), django__django-11019 (16; nobody solves it), django__django-11564 (13, twelve of them editing the wrong file), django__django-15819 (13: every agent ran the full inspectdb test module, but the hidden test fixture changes the module's expected output). These tasks are failed by all three blog harnesses at similar rates; they are a property of the models and the benchmark, not of Sorcar.

3.5 Sorcar-specific artifacts

4. Where the money goes

The per-call usage events in sorcar.db give the token split of every model call (uncached input, cache reads, output). Prompt caching works: 85–96% of prompt tokens were served from cache for every model. The cost is therefore driven by two things: output tokens (thinking plus text), which are 31–60% of spend, and the sheer volume of context re-read on every turn, which even at one tenth of the input price is 23–48% of spend for the long-running models.

5. What would move the numbers

The recommendations are ordered by the size of the effect the data supports. None of them uses knowledge of the benchmark tasks; all of them are harness or wrapper changes that apply to any long-running coding job.

6. Data and reproducibility

• Trial records: benchmarkings/harnesstax/results/baseline/summary.json (1,260 records) and per-trial trajectory.jsonl / result.json / model.patch.
• Official grading: SWE-bench _eval/*/logs/run_evaluation/*/report.json; Harbor verifier/test-stdout.txt and result.json.
• Daemon events: ~/.kiss-harnesstax/sorcar.db (1,518 tasks, 2,248,304 events; usage_info per call, tool_call/tool_result/thinking_delta with timestamps).
• Analysis scripts used for this report: tmp/analysis.py, tmp/cache_analysis.py, tmp/build_report.py (outputs in tmp/analysis_out.txt, tmp/cache_out.txt).
• Blog reference values: benchmarkings/harnesstax/data/blog_tables.json, blog_per_task.json (from harnesstax.github.io).