Review of "Runtime Placement and Evidence in LLM Agent Harnesses" (ACL 2027)

Summary
The paper surveys the model-external runtime of LLM agents and asks where a runtime decision acts and what evidence licenses a claim about it. It arranges 123 records, 20 code audits, and 35 appraised studies under a six-phase lifecycle and releases three untested specifications. The appraisal is the real contribution; the lifecycle is a tidy map of familiar ground.

Strengths
- Sec. 6.2's thesis is right: a gate after dispatch is an audit; a verifier that reports only to the scorer cannot repair.
- The appraisal separates clear from partial reporting and shows how much hangs on that threshold (sealed test 1/33, uncertainty 0/33).
- Each audit names the exact surface (Claude Code hooks, LangGraph interrupt(), Inspect approval scopes) and a falsifying test.

Weaknesses
- The bibliography contains no 2025 paper. It jumps from the 2024 benchmarks to 2026 preprints, skipping the year that produced most of the evidence it wants.
- Tables 8 and 16 name some 25 works (Toolformer, ToolLLM, MemGPT, Voyager, Reflexion, AutoGen, MetaGPT, CRITIC) that have no reference entry.
- One author coded everything; no agreement statistic. The blind re-audit packet is released but never used.
- Figure 10's three "counterexamples" are drawings, not runs.

Detailed review
Novelty is moderate. At least five harness surveys exist already (Meng, Li, He, Guo, Wei), and the lifecycle is Gray and Reuter's transaction model with a MAPE-K loop attached. Appraising harness claims by study design is new. Keep that and cut the rest by a third.

The missing year matters. Kapoor et al. (2024, "AI Agents That Matter") made the cost-controlled and holdout argument that Sec. 5 rediscovers. Zhu et al. (2025, Agentic Benchmark Checklist) documented oracle flaws in SWE-bench Verified and tau-bench; tau-bench's pass^k is an uncertainty metric in use since 2024, so "clear in 0" needs a qualifier. For Commit, CaMeL (Debenedetti et al., 2025), AgentSpec (ICSE 2026, arXiv March 2025), and Progent are the pre-effect gates to cite. Agentless (2024) is the cleanest configuration comparison and belongs in the n=9 family. For traces, the AgentOps taxonomy (Dong et al., 2024) and the OpenTelemetry GenAI agent-span conventions already define most of Table 20; map to them. For Tables 13-14, MAST (Cemri et al., 2025) coded 1,600 traces at kappa 0.88. That is the bar.

Make the thesis testable. Figure 10 describes three experiments cheap to run on AgentDojo or AppWorld: the same approval check before vs after dispatch; the same verifier sent to the scorer vs the next turn; the same checkpoint with vs without effect markers under a killed process. One controlled contrast would carry more weight than 123 records.

Prose: the main text says what it is not many times and uses "rather than" many times. Put statements in positive form. "Inspectable" appears many times and means little by page 3.

Fix the references, run the three experiments, add a second coder, and this is a good Findings paper.
