# Study Design Reviewer Feedback

The manuscript provides a comprehensive and foundational framework for characterizing coded modulation schemes as coset codes, successfully unifying lattice codes and trellis codes under a single geometrical classification. To ensure the robustness of the comparative claims and the proposed "folk theorem," I recommend validating the heuristic performance metrics and expanding the discussion on complexity dependencies.

# Comments

1. Validation of the Effective Coding Gain Heuristic: The comparison of code performance (Table V, Table XI, and Figure 12) relies heavily on the "rule of thumb" that a factor of two increase in the error coefficient  $(\mathcal{N}_0)$  corresponds to a 0.2 dB loss in coding gain. While this linear approximation allows for rapid comparison, it presumes a specific slope of the error probability curve that may not hold for all code classes or at all signal-to-noise ratios (SNRs). I recommend including a rigorous validation of this heuristic—such as a Bit Error Rate (BER) simulation or an analytical error bound comparison—for at least one representative code from the new classes (e.g., Class VI or VIII) to confirm that the heuristic accurately preserves the ranking of these codes against established benchmarks.

2. Dependency of Complexity Metric on Implementation: The "normalized decoding complexity" metric $(\mathcal{N}_D)$ is defined based on the specific trellis-based decoding algorithms to be presented in Part II. Consequently, the performance-vs-complexity trade-offs (Figure 12) are tightly coupled to this specific implementation strategy. To support the generality of the geometrical classification, I suggest the authors discuss how the ranking of the proposed code classes might change if alternative decoding architectures (such as parallelized decoding or sub-optimal reduced-state decoding) were employed. If the complexity advantage is implementation-specific, this limitation should be explicitly stated in the conclusions.

3. Impact of Higher-Order Distance Terms: In Section VII, the discussion and the "folk theorem" rely almost exclusively on the minimum squared distance $(d_{min}^2)$ and the immediate error coefficient $(\mathcal{N}_0)$. The influence of next-nearest neighbors $(\mathcal{N}_1$, etc.) is largely dismissed. However, for the proposed high-state codes (e.g., 256 states), the density of the lattice at distances slightly greater than $d_{min}$ could potentially lead to performance degradation that is not captured by $\mathcal{N}_0$ alone. I recommend reporting the $\mathcal{N}_1$ values or the next spectral line for the "best" proposed codes (Class VIII) to demonstrate that the error coefficient does not grow pathologically at the next distance shell, ensuring the validity of the effective coding gain claims.

4. Justification of the "Peripheral" Nature of Shaping: The introduction posits that constellation shaping is "peripheral" and has only a minor effect on overall coding gain. While this allows for the separation of coding and shaping, it may oversimplify the comparison when applied to the specific finite constellations used in the referenced modem standards (e.g., V.32 series). I suggest the authors clarify the limits of this assumption: specifically, does the "peripheral" nature of shaping hold for the smaller constellations often required in lower-rate fallbacks, and does ignoring shape gain introduce a bias when comparing the proposed coset codes against schemes where shaping and coding are more integrated?

5. Logical Basis of the "Folk Theorem": The "folk theorem" (suggesting a specific dB gain per doubling of states) is presented as a major empirical conclusion. To strengthen the logical support for this claim, I recommend connecting this observation to the theoretical sphere packing bounds or capacity limits for the specific dimensions discussed (N=2, 4, 8, 16). Explicitly comparing the slope of the "folk theorem" line to the theoretical diminishing returns dictated by channel capacity would determine whether this theorem is a fundamental property of the channel or a limitation of the current code construction methods.

<!-- PAGE BREAK -->

# Reproducibility Reviewer Feedback

The manuscript provides a comprehensive geometrical classification of coset codes and introduces several new classes of trellis codes, offering a unified framework for lattice and trellis-coded modulation. However, the reliance on unpublished data for key performance comparisons and the use of a heuristic "rule of thumb" for performance estimation without validation via error rate simulations significantly limit the reproducibility and rigorous validation of the proposed "folk theorem."

## Comments

1. Reliance on unpublished data for key performance metrics. Table V and Figure 12 present performance parameters ($\mathcal{N}_0$, $\tilde{\mathcal{N}}_1$, $\tilde{\mathcal{N}}_2$) and effective coding gains for Ungerboeck-type codes based on "results of Eyuboglu and Li (unpublished)." Because these results are central to the comparative analysis and the claims regarding the performance-complexity trade-off, they must be verifiable. Please incorporate the methodology and data used to derive these coefficients into the manuscript (or a supplement) or restrict comparisons to published, reproducible datasets.

2. Validation of the "Rule of Thumb" for effective coding gain. The manuscript relies heavily on a heuristic formula to calculate "effective coding gain" ($\gamma_{\mathrm{eff}}$), subtracting approximately 0.2 dB for every factor of two increase in the error coefficient $\mathcal{N}_0$. This approximation assumes a specific slope of the error probability curve at error rates of $10^{-6}$. However, this metric considers only the nearest neighbor count ($\mathcal{N}_0$) and ignores the impact of the complete distance spectrum (next-nearest neighbors, etc.), which may vary significantly between the different code classes (e.g., Class I vs. Class VIII). To ensure statistical rigor, please provide evidence (such as error bounds or simulation points) demonstrating that this heuristic holds with sufficient accuracy across the diverse new code classes introduced, rather than assuming a uniform penalty applies to all.

3. Absence of error rate simulations to support theoretical claims. The conclusions, including the proposed "folk theorem" regarding the number of states required for specific gains, are based entirely on the derived geometrical parameters and the aforementioned heuristic. To validate these theoretical projections, please include Bit Error Rate (BER) vs. SNR simulation curves for representative codes from the newly proposed classes (Classes I-VIII). This is necessary to confirm that the asymptotic coding gain  $\gamma$  and the adjusted  $\gamma_{\mathrm{eff}}$  accurately reflect performance in the target error rate regime ( $10^{-6}$ ), particularly for codes with high state complexity where error propagation or path multiplicity might deviate from the model.

4. Sensitivity of the complexity comparison. The "normalized complexity" metric  $(N_{D})$  used in Figure 12 and throughout the text is defined based on specific trellis-based decoding algorithms presented in Part II. The author acknowledges this metric is "highly implementation-dependent." To ensure a fair benchmarking comparison, please clarify whether the baseline "conventional Viterbi algorithm" complexity was optimized to the same degree as the lattice decoding algorithms. If the complexity metric favors the proposed lattice structures due to algorithmic choices rather than fundamental computational bounds, the comparative advantage shown in Figure 12 may be overstated.

5. Incomplete specification of high-state code generators. While the general construction of Class V through Class VIII codes is described via Figure 13 and textual descriptions of the linear circuit  $T$ , the specific wiring or generator polynomials for the high-state versions (e.g., the 128-state and 256-state codes in Table XI) are not explicitly tabulated. To ensure these specific code instances are reproducible by other researchers, please provide the exact generator matrices or polynomials used to calculate the parameters listed in Table XI, rather than relying on the generic construction description.

<!-- PAGE BREAK -->

# Limitations & Context Reviewer Feedback

The manuscript provides a comprehensive and unifying framework for coded modulation through the concept of coset codes, successfully categorizing a wide array of existing schemes. However, to meet the standards of a high-impact publication, the authors should more rigorously define the boundaries of their performance metrics and further contextualize the "folk theorem" proposed in the conclusion.

# Comments

1. Justification of the "Effective Coding Gain" Heuristic: The manuscript relies on a "rule of thumb" that every factor of two increase in the error coefficient  $(\mathcal{N}_0)$  reduces the coding gain by approximately 0.2 dB. While this facilitates comparison, it is an empirical approximation valid only at specific error rates (cited as  $10^{-6}$ ). The authors should discuss the limitations of this heuristic, particularly how it degrades at different SNR regimes or for codes with steep versus shallow error probability slopes. A more robust justification or a discussion of the sensitivity of the results to this approximation is necessary for the performance comparisons to be fully generalized.

2. Dependency of Complexity Metrics: The comparison of codes based on "normalized decoding complexity"  $(\widetilde{\mathcal{N}}_D)$  is heavily dependent on the specific trellis-based decoding algorithms derived in Part II. The authors should acknowledge that this metric may not represent the fundamental lower bound of decoding complexity for these codes, but rather the complexity relative to a specific implementation architecture. A discussion on how alternative decoding approaches might alter the relative ranking of the "Class" codes versus the Wei or Ungerboeck codes would strengthen the generalizability of these findings.

3. Limitations of the "Shape Gain" Decoupling Assumption: The Introduction states that constellation shaping is "peripheral" and "almost independent" of the fundamental coding scheme. While theoretically sound for infinite lattices or large constellations (high rate), this assumption weakens for small constellations where boundary effects are significant. The authors should explicitly discuss the limitations of the coset code framework when applied to systems with low spectral efficiency or small signal sets, where the interaction between the coset selection and the constellation boundary (shape gain) may not be negligible.

4. Scope of the "Folk Theorem": The conclusion proposes a "folk theorem" relating the number of states $(2^{u})$ to the fundamental coding gain (e.g., 256 states for 6 dB). The authors should clarify whether this observation is believed to be a fundamental limit of the coset code structure itself, or merely a limitation of the currently known binary construction techniques. Contextualizing this against the potential of non-binary (e.g., ternary) codes mentioned briefly would add significant value to the discussion of future directions.

5. Asymptotic Nature of Fundamental Coding Gain: The parameter $\gamma(C)$ is essentially an asymptotic figure of merit (high SNR). The manuscript would benefit from a discussion on the limitations of using $\gamma(C)$ as the primary design criterion for channels operating at lower SNRs, where the error coefficient and nearest-neighbor multiplicity play a more dominant role than the asymptotic gain suggests.

6. **Distance Invariance Assumption**: The manuscript notes that "All codes in this paper are distance-invariant." To ensure completeness, the authors should briefly discuss whether the coset code framework inherently guarantees distance invariance, or if it is possible to construct valid coset codes that lack this property. If non-distance-invariant coset codes are possible, the limitations of the proposed analysis methods (which rely on the all-zero sequence) should be addressed.