Review: Paper 3798 - Pseudo-Generator: Exposing Parameter Generalization Failures in LLM-Generated Chisel Hardware

Summary
-------
The paper names a failure mode, the Pseudo-Generator: a Chisel generator that passes at its default parameters but hard-codes depths, widths, or bounds and breaks at other legal values. ChiselGenBench (38 tasks, 280+ sweep configurations) measures it; 25-56% of default-passing generators from five LLMs fail the sweep. A 2x2 ablation on three models argues that a generation plan raises default pass rate while counterexample repair raises sweep pass rate.

Strengths
---------
+ The question is right. Single-configuration testing of generator code is a blind spot, and the sweep (default, minimum, non-power-of-two, stress) is a clean instrument.
+ PGR is an honest metric; the paper flags its denominator problem itself.
+ The plan/repair separation is a useful experiment.

Weaknesses
----------
- Repair is scored on the configurations it was shown. The failing sweep points go to the model, and the same points define PSPR. That is patching to the test set (Smith et al., FSE 2015); nothing stops `if (depth == 3)`.
- No resample baseline. Repair gets up to three more calls plus the verifier; direct gets one. Compare with best-of-4 direct samples under the same verifier.
- Section V.G counts 20, 17, and 10 Pseudo-Generators at direct_repair round 1 for Qwen-2.5, DeepSeek-V3, and GPT-4o. Table IV's direct runs, the same code at temperature 0, yield 5, 6, and 4. Both cannot be right. And a type error cannot be "triggered only under swept configurations"; Scala compiles once.
- The Fix column is not (fixed)/(previously failing) throughout: Qwen 4/34 = 11.8 and GPT-4o 2/26 = 7.7 work, but DeepSeek repair is 6/27 = 22.2, not 25.0, and GPT-4o plan+repair 7/27 = 25.9, not 23.1.
- McNemar p = 0.031 for GPT-4o means six one-sided discordant tasks; Table V shows PSPR rising 11 to 18 with "none regress", which is seven (p = 0.016). The pooled 16 also disagrees with 3+7+7.
- The plan's DPR jump (DeepSeek 44.7 to 89.5) with a PSPR drop is unexplained. My guess: the plan fixes port names and types so the ChiselTest harness compiles, not the logic. Report default failures by stage.
- Related work misses the closest systems. ChatChisel (Liu et al., ISEDA 2024) and ReChisel (Niu et al., DAC 2025) generate Chisel with decomposition and compile/simulate reflection loops; AutoChip (Thakur et al., 2023) and VerilogCoder (Ho et al., AAAI 2025) do the same for Verilog. CVDP (Pinckney et al., 2025) has parameterized Verilog tasks. Fu et al. (arXiv 2604.17097, April 2026) evaluate Chisel among six IRs with a repair stage.

Detailed comments
-----------------
Novelty is moderate. Hard-coded internals behind a parameterized signature is a failure every hardware reviewer knows; the contribution is to name it, count it, and show the count survives model scale. That is worth publishing once the measurement is trustworthy. The plan-then-repair loop is not new, and the Chisel-versus-Verilog framing is weak: Verilog parameters and generate blocks admit the same sweep. Run the 38 tasks in SystemVerilog with parameter overrides; if PGR is similar, you have a language-independent result and a better paper.

Fixes, in order of payoff:

(1) Hold out one boundary and one stress configuration per task that repair never sees, and report PSPR on them. That is the number the title promises.

(2) Add the best-of-k resampling baseline with the same verifier and call budget.

(3) Use the dependency graph you define. Elaborate at two configurations and diff the FIRRTL: if a memory depth or register width the parameter governs does not change, propagation failed. That is a cheap static Pseudo-Generator detector and a real contribution.

(4) Reconcile V.G with Table IV, define Fix, and recount the McNemar cells.

(5) Ablate on Qwen-3.6-Plus, your strongest model.

(6) State that every golden reference passes every sweep configuration, and who checked that the parameterized testbenches are not themselves Pseudo-Generators.

(7) Temperature 0 on the GPT-4o and DeepSeek APIs is not deterministic. Three runs would show whether a two-task PSPR change (GPT-4o repair) means anything.

Smaller things: "desensitized" should be "redacted." Cite McNemar directly, not through Arcuri and Briand. Table IV gives DeepSeek PSPR as 29.0; 11/38 is 28.9, as Table V has it.
