RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery

This paper argues that once an LLM finder can edit source, mutate state, and script a browser, "the payload executed" is no longer proof of an exploit, and it builds a verifier designed to survive exactly that finder. The problem is timely and the mechanism is coherent. My concerns are about what the current experiments actually establish. Several headline sentences claim more than the reported data supports, one figure caption contradicts the paper's own text, and the strongest missing evidence is a direct security analysis of the verifier itself rather than another model.

1 · Summary

An LLM coding agent explores a target web application with white-box access (shell, source, seeded credentials) and proposes candidate XSS. Each candidate is resubmitted as a structured proof of concept: an attacker HTTP script plus a sequence of typed victim browser actions over CSS selectors. RECEIPT restores a clean application snapshot, runs the attacker script from a sandboxed remote client under the attacker role, drives a separate victim browser under the victim role, and accepts the finding only if a Chrome DevTools Protocol sentinel — seeded with a fresh single-use token handed to the attacker script — fires in the victim page. Four mechanisms carry the argument: environment isolation, PoC constraints, role separation (with diverged credential seeding), and verdict binding.

Reported results on 95 apps (median 13.1k stars, seven stacks, $20/target): Table I reports 30 confirmed findings on latest releases (24 previously unknown, 12 acknowledged/patched), all 30 counted true positive; the same agent self-judging reported 27 with 10 true. Table III reports 49% detection and 36% labeled-CVE recovery on 45 known-vulnerability targets with 0 false positives at target level. Table IV ablates on 27 targets, with precision moving 45% → 9% → 19% → 44% → 100% as browser binding, isolation, PoC constraints, and role separation are added.

2 · Novelty: real, but the delta must be stated more carefully than it is

The general strategy — reject the LLM's self-report and require an executable, deterministic proof — is not new in 2026. Execution-grounded XSS validation runs through Black Widow and the black-box lineage the paper cites, and Spider-Scents already separated "candidate output" from "vulnerability" from "exploit" for stored XSS (leaving exploitability to manual analysis). Two recent systems verify LLM-agent findings with executable oracles: Revelio (memory safety; cited by the paper as [23]) and Antiproof (proof-of-exploitability oracles across a broad class). RECEIPT must situate itself against both, and precisely.

On Revelio: the paper's implicit contrast overstates the gap. Revelio already runs in a container, re-executes an independent proof-of-vulnerability in a fresh subprocess, uses an existing external-input harness, and states rules that exclude source modification, internal-state mutation, private APIs, and privileged local access. So the open question RECEIPT should claim is not "prior work ignored contamination" but "are those exclusions technically enforced against an agent that can rewrite the repository, and what does enforcement require for a browser-delivered vulnerability?" RECEIPT's defensible, narrower delta is the two-principal browser semantics: verifier-controlled victim identity, a typed (non-code) victim-action channel, credential divergence, and a browser-side verdict bound to a per-attempt token. Claim those; do not claim generic isolation or "constrained executable evidence" as uniquely RECEIPT's.

On Antiproof: it is technically closer than the paper (which does not cite it) will want to admit — a network-facing prover, separated target and oracle, a cryptographically random challenge, and acceptance tied to a concrete attacker capability under a deployment trust boundary. Those parallel RECEIPT's attacker sandbox, token, trusted verifier, and exploitability semantics. Note, however, that Antiproof's first arXiv version is dated 14 July 2026, after the 30 June 2026 ICSE deadline. It is therefore concurrent work: it cannot count against the submission's originality, and it would be inappropriate to fault the authors for not citing it. The correct ask is a paragraph in the revision acknowledging it as concurrent and distinguishing the designs (XSS-specific attacker/victim browser split and typed benign-user interaction versus Antiproof's broader oracle).

One process note, not a technical one: the review copy is double-anonymous, so the novelty argument must be made purely on cited artifacts and stated mechanisms. Any reasoning that leans on guessing which group wrote the paper should be removed from the discussion; it does not bear on the technical delta and it is out of bounds for this track.

A feature comparison would do more than a scatterplot

Rather than an invented one-dimensional "power" axis, I recommend the authors include a table of the properties that actually differ across systems, with citations. My reconstruction:

3 · The reward-hacking taxonomy is well-observed; it needs coding, not just narration

C1 (state/source contamination), C2 (verifier self-injection), C3 (threat-model mismatch) is a useful decomposition, and Figure 2 of the paper is its clearest exhibit. Section II sources the taxonomy to "a preliminary study," but this undersells the paper's own evidence: RQ3's browser-verdict configuration produced 111 accepting reports, and the discussion attributes them to these behaviors. The fair criticism is therefore not "anecdote" — it is that the paper never codes and counts the 111 (and the other configurations') failures by category. I would ask for a mutually exclusive coding protocol, per-category counts (contamination / self-injection / role mismatch / other), representative transcripts in an appendix, a second coder, and a disagreement-resolution step.

4 · The verifier's own attack surface is the missing analysis — this is the main revision condition

The paper's central promise is that a capable finder cannot forge acceptance. That is a security claim about the verifier, and the paper argues it by construction but does not test it adversarially. Before I believe "no false positives," I want the authors (or the artifact) to establish, with a malicious-PoC bypass suite, that the following are actually closed:

• Victim URL handling. Strict scheme/origin checks and redirect handling on the submitted victim URL (no data:/javascript:/file:, no open-redirect into an attacker origin, credential/parameter stripping verified).
• Control-plane isolation. CDP endpoint, the sentinel callback, browser debugging ports, and verifier services must be unreachable from the attacker container and from the target app (including via SSRF or DNS rebinding into verifier services).
• Reset pristineness. Source, databases, volumes, caches, and any external services are truly restored per attempt — not just the primary container.
• Credential hygiene. Undocumented default credentials, leftover API keys, and pre-existing sessions cannot substitute for the diverged-seeding that role separation relies on.
• Typed-action expressiveness vs safety. Exactly what data is permitted in fill, select, wait, and selector fields, and a demonstration that none of these can smuggle executable content into the victim context.
• Sentinel frame binding. The sentinel must be bound to the intended top-level victim frame and navigation, not fired by an unrelated frame or a stray eval.

A demonstrated bypass suite (attempted C1/C2/C3 attacks plus these boundary cases, all rejected) would do more for the paper's soundness than any number of additional discovery targets. Running a second model, by contrast, mostly tests the finder, not the verifier; make it a desirable external-validity experiment, not the primary evidence of soundness.

5 · The precision claim overreaches its sample

The paper says in places that RECEIPT "eliminates all false positives." What the data supports is narrower: RQ1 reports 30/30 accepted findings judged true; the full-configuration ablation reports 14/14; RQ2 reports 0 false positives at the target level (Table III), which is not the same as report-level precision. Three fixes:

• Report the number of unique accepted PoCs / reports / distinct root causes per RQ, with overlaps between the RQ1 set and the ablation subset made explicit. (The ablation is on a subset of the same 50 applications, possibly with repeated runs, so the 30 and the 14 cannot simply be added — please state the true unique count.)
• Give the adjudication protocol: who judged, against what independent criterion, with a second adjudicator and an agreement statistic on the true/false and role/CVSS labels.
• Attach exact binomial confidence intervals and phrase the result as "no false positives were observed in this evaluation," which the paper often already does, rather than implying a finite sample proves zero population error. The specific sentences asserting elimination of all false positives are the ones to soften.

6 · The ablation is informative but not a clean component attribution

Table IV is the paper's most interesting result, and it should be presented with all three quantities, because precision alone hides the most important fact: going from self-judgment to a browser-bound verdict doubles true positives (5 → 10) even as precision collapses (45% → 9%). "Worse than nothing" is therefore true for precision only, not for detection — the browser signal opens many more accepting trajectories, most of them resting on privileged state manipulation. Please state it that way and drop any language implying the agent "learns" to game the check over time; Table IV shows outcomes, not a temporal learning curve.

Two design caveats the authors should acknowledge: these are stochastic end-to-end agent runs, so each verifier configuration changes the agent's trajectory — this is not a replay of one fixed candidate set through five checkers. And a cumulative order cannot attribute a component's marginal effect independently of interactions. A cleaner complement: submit one common corpus of genuine and adversarial PoCs to each verifier configuration and report accept/reject directly. That isolates the verifier from the finder.

7 · Repeatability of the replay itself

The paper scopes "deterministic" to a fixed snapshot, threat model, and PoC, which is the right scope; I found no place where it overclaims system-wide determinism, so I withdraw that as a concern. The sharper question the title invites: is browser replay actually repeatable given asynchronous rendering, timers, network scheduling, service workers, background jobs, randomized application data, and wait actions? Please replay each accepted PoC several times from reset and report acceptance stability, timeout settings, and any flaky cases. Verifier replay stability is more central to the contribution than discovery-run variance (though the latter is worth a line for external validity).

8 · Baselines and generalization

Two clarifications and one correction to my own first read:

• Correction: Table I already separates reachable targets (ZAP 31/50, YuraScanner 26/50, Black Widow 22/50) from reports, and the text states scanners produced zero true positives even on apps they could crawl. So "reachable" and "reached-but-empty" are not blended, as I initially thought. What is still needed is each scanner's version, configuration, authentication setup, crawl budget, and per-target outcome — plus a known-positive smoke test (run each integrated scanner on a few targets it should detect) so a zero on the unknown suite is distinguishable from a broken integration.
• The black-box comparison does not isolate RECEIPT's verifier contribution: those tools differ in source access, crawling, workflow handling, and output semantics, and ZAP's "potential-XSS alerts"/YuraScanner's "HTML-injection candidates" may not be intended as final confirmed reports. Counting each candidate as an equivalent false report needs justification. The same-agent configurations (self-judge vs. full RECEIPT) are the relevant verifier comparison; the scanners primarily support a reach/discovery comparison. Say so explicitly.
• The recall generalization is too strong. Recovering 16 labeled CVEs across 45 curated targets does not support "more than one-third of all known XSS vulnerabilities" without a sampling frame. State how the 45 targets / 34 apps were selected, which candidates failed to build, and whether inclusion depended on packaging difficulty or expected discoverability; then report the result as 36% recovery on this benchmark. The post-cutoff/no-network protocol is a real strength — foreground it — but give evidence for the claimed August-2025 model cutoff and note that vulnerable code and issue threads can predate CVE disclosure.

I would not make recreating YuraScanner's 20-app benchmark or running CVE-Bench a blanket requirement; the paper reasonably argues its two-principal semantics do not map cleanly onto them. Ask instead for a compatibility table and, if the victim-role semantics can be represented, one calibrated shared subset.

9 · Completeness / false negatives of the constrained PoC language

Almost all of the evaluation is about precision. The flip side deserves attention: the fixed victim-action vocabulary may be unable to express genuine XSS that needs multi-window interaction, drag-and-drop, clipboard operations, file selection, browser permission prompts, OAuth redirects, or WebSocket-driven rendering. The 23/45 recovery misses are attributed broadly to "exploration-side limitations," but some may be verifier-language false negatives. Please split the misses into: candidate never found; candidate found but no PoC constructed; real PoC rejected or inexpressible in the action language. That distinction tells a reader whether to push on the finder or on the verifier interface.

10 · Role manifests

Role separation is only as sound as the manifest. This is a soundness assumption, not a presentation detail, so I want provenance: how many of the 95 manifests came from explicit security documentation versus reviewer judgment, an inter-rater agreement figure on role and trusted-author labels, and a sensitivity check for a wrong manifest. Note that a same-role row is not automatically invalid: Admin → Admin or User → User can be a legitimate cross-principal attack when two distinct accounts exist and the authoring feature is not documented as trusted; Admin → User can be valid in a multi-tenant setting. The label to interrogate is principal separation and documented feature-level trust, not the word "Admin." For the Admin → rows in Table II specifically, state the two account identities and why the feature is not a trusted-author feature.

11 · Smaller points

• The Docmost example (Figs. 4/5) is the paper's best exhibit; keep it prominent. I withdraw a suggestion from my first pass — the text does explain that traversal causes an uploaded SVG to be served inline as a top-level document and execute, and there is no open redirect in the chain.
• Typo: "malicious SVG attachmentch" in the Docmost walkthrough.
• Surface the "found a different real bug, not the labeled CVE" row (6/45): it is quiet evidence the tool is not merely rediscovering disclosed bugs, and it strengthens the discovery story.
• Browser scope: since replay is real Chromium, CSP, framework auto-escaping, and mutation XSS are largely target behaviors the browser will observe naturally — these are not unexplained sentinel assumptions, and mobile WebViews are already declared out of scope. If a scope paragraph stays, pin down the Chromium version, headed vs. headless differences, and frame policy.
• Cost: "$20/target over 5 hours" mixes budget and wall-clock. Report token cost, compute cost, human setup labor, and wall-clock separately, and give a cost-per-confirmed-bug figure.
• ICSE 2027 is IEEE format this year and runs a reference-integrity check that desk-rejects fabricated or unverifiable citations. Several 2026 references (including recent preprints) should be double-checked to resolve to findable records. The anonymized 4open.science artifact is the right call; verify the disclosure wording in §V does not de-anonymize via named maintainer threads.

12 · Conditions for acceptance

1. acceptance condition

   State the verifier's threat model and test it adversarially

   Explicit trust boundary plus a malicious-PoC bypass suite covering C1–C3 and the boundary cases in §4 (victim-URL scheme/redirect, CDP/port isolation, SSRF/DNS-rebinding, reset pristineness, default credentials, typed-action data, sentinel frame binding). This is the paper's soundness evidence.

2. acceptance condition

   Account for unique reports and adjudicate independently

   Per-RQ unique accepted PoCs/reports/root causes with explicit overlap; second adjudicator with an agreement statistic; binomial CIs; replace "eliminates all false positives" with "none observed in this evaluation."

3. acceptance condition

   Show the replay is repeatable

   Replay each accepted PoC multiple times from reset; report acceptance stability, timeouts, and flaky cases.

4. acceptance condition

   Better-controlled ablation and baseline calibration

   Report reports/TP/precision together; add a common-corpus PoC experiment across verifier configurations; publish baseline configs and a known-positive smoke test for each scanner.

5. strengthens

   Situate against Revelio (narrow the delta) and Antiproof (concurrent)

   Feature table with citations; one paragraph each; state what enforcement a browser-delivered vulnerability requires that a sanitizer-checked PoV does not.

6. strengthens

   Code and count the reward-hacking behaviors

   Mutually exclusive categories over the 111 (and other) accepting reports; counts, traces, second coder.

7. strengthens

   Separate the recall misses by cause

   Never-found vs. no-PoC vs. inexpressible-in-action-language; this locates the bottleneck.

8. external validity

   A second finder model; manifest provenance; a calibrated shared subset

   Useful, but secondary to the four conditions above.

13 · What the paper already gets right

Stated soberly: the separation of untrusted exploration from constrained replay is the correct architecture for this threat; the attacker-to-victim acceptance semantics are precise and, on the role-enforcement axis, appear to be the paper's own contribution; evidence is grounded in real-browser execution; the post-cutoff and runtime-leakage controls are a genuine methodological strength; the target set is broad and realistic; the accepted findings ship as replayable artifacts for maintainers; and the paper is candid that the sentinel alone is insufficient and that detection differs from labeled-CVE recovery. Twenty-four disclosures with twelve acknowledged or patched is real-world impact. These strengths are why the recommendation is Major Revision rather than reject.

Sources consulted for this report. Peer-reviewed / archival: Antiproof (arXiv 2607.12316, concurrent), Revelio (arXiv 2606.22263), CVE-Bench (arXiv 2503.17332, ICML'25), Spider-Scents (USENIX Security'24), YuraScanner (NDSS 2025), and cross-checks against the paper's cited bibliography (Black Widow, Black Ostrich, Krakovna/Skalse on reward hacking, Panickssery on self-evaluation, Heiderich on mXSS). Non-archival context: the ICSE 2027 Research Track CFP, the XBOW company blog, the Irregular/Wiz web-security-agent write-up, and a Google Scholar/SERP survey of 2026 reward-hacking-in-agentic-security work. These last four are industry/context sources, not peer-reviewed. I did not build or replay the paper's artifact for this report.

Referee report · ICSE 2027 program committee · confidence 3/5 · expertise: program analysis, web security, LLM agents · this report was itself meta-reviewed and corrected before submission.