Review of Submission 5091 (Design-Specification Tiling for ICL-based CAD Code Generation)

Summary

DST picks k in-context exemplars for text-to-CadQuery generation by greedily maximizing the weighted fraction of the query's word n-grams that appear in the union of the exemplars' descriptions. Coverage is monotone submodular, so greedy gives the usual (1-1/e) bound. On 900 Text2CAD problems in three difficulty tiers, DST beats Random, Levenshtein, BM25 and a k-means diversity baseline for three LLMs, by a wide margin on Qwen3 and by a hair on the other two.

Strengths

- Exemplar selection for CAD code generation is untouched; I found no prior work.
- The worst-case penalty for invalid outputs and the 96-transform alignment search are sensible and clearly described.
- Fig. 6 shows the failure mode of similarity retrieval convincingly.
- Code is released.

Weaknesses

- The method is CAST [13] with n-grams in place of AST subtrees, and the paper does not say so.
- Set-BSR, Cover-LS and CEIL, the coverage-selection literature, are absent.
- The released code differs from the paper in the n-gram sizes, preprocessing, stopping rule, reported tiling ratio, and the LLM used.
- Baselines appear to retrieve from a different database than DST.
- The strongest baseline (CodeBERT top-k) is left out of Table II.
- One run, no seeds, no significance tests; most DeepSeek and Claude gains are within 0.01.

Detailed review

Novelty. Reference [13] (TSE 2025) proposes a coverage surrogate, calls its maximization NP-hard, proves it submodular, and runs greedy with the (1-1/e) guarantee. Sections II-B and III repeat that argument with word n-grams. Gupta, Gardner and Singh (EMNLP 2023) optimize set-level BERTScore-Recall the same way; Levy, Bogin and Berant (ACL 2023) cover output structures; Ye et al. (ICML 2023) use DPPs. None is cited, and Proposition 1 spends a column proving that a coverage function is submodular. State plainly that DST is an application of set-coverage selection to CAD, cite these papers, cut the proof to a sentence, and run Set-BSR as a baseline. The CAD application is the contribution, so make it the paper.

Paper versus code. coverageICL.py uses n in {1,2,3,4,8,16,32}, not {2,4,8,16,32}; strips stopwords; splits on periods so n-grams never cross sentences; uses the "intermediate" prompt level; and when no candidate adds coverage it resets the uncovered set and keeps going, instead of Algorithm 1's break. The coverage it reports is the unweighted fraction of n-grams, so Fig. 5 does not plot Eq. 6. The README names Qwen3-Coder-30B-A3B-Instruct; the paper says Qwen3-30B-A3B. Make the text match the code, and say which model ran.

Baselines. baselineICL.py loads the exemplar pool from a hard-coded low75database.csv while DST reads the full database; LDSIM and BM25 pre-filter candidates by a keywords column the paper never mentions; LDSIM is word-level Levenshtein on descriptions, not "code similarity"; Random samples with replacement. Rerun every method on one pool, unfiltered. CodeBERT top-k beats DST on Easy in VSR, CD and ECD (Table III) and belongs in Table II.

Soft matching. The code consults embeddings only for candidates with zero exact overlap, so soft matching barely changes anything. The "numerical noise" story in RQ5 is wrong; drop it or implement the variant properly.

Boilerplate. Text2CAD descriptions are LLM-generated from templates, so 16- and 32-grams weighted 16 and 32 mostly reward shared filler. Show the covered 32-grams for a few queries, ablate over N, and try IDF weights. DeepCAD has many near-duplicate shapes: report a "copy the top exemplar's code" baseline and the IoU between each query's ground truth and its best exemplar's, to show DST is not retrieving the answer.

Statistics and data. 300 problems per tier, one sample. DeepSeek Middle IoU rises by 0.003 and Claude Easy IoU falls; nothing in those blocks is shown to be real. Use three seeds and paired bootstrap over problems. filter_data.py, now commented out, built tiers of 40K/60K/50K and dropped every abstract containing "L-shaped"; the paper claims equal thirds. Cite Xie and Ju (Text-to-CadQuery, May 2025), the likely source of the CadQuery annotations, and compare against one of their fine-tuned 7B models, since the introduction argues fine-tuning is impractical. Fig. 5 pools five shot counts; compute the correlation per query at fixed k. Report selection latency; the code measures it.
