# Coset Codes—Part I: Introduction and Geometrical Classification

**Date**: 03/08/2026
**Domain**: unknown
**Taxonomy**: academic/research_paper
**Filter**: Active comments

---

## Overall Feedback

Here are some overall reactions to the document.

**Outline**

This review identifies methodological gaps in the foundational treatment of coset codes, including deferred proofs that compromise standalone rigor, unverified assumptions of geometric uniformity for nonlinear trellis codes, and an overlapping classification scheme. The paper's reliance on infinite lattice approximations without quantifying finite constellation effects, combined with an unjustified rule of thumb for effective coding gain calculations, limits the practical applicability of the performance comparisons.

**Deferred Proofs for Critical Lattice Decomposition Results**

The paper presents Lemma 2 in Section II-F as establishing the existence of Ungerboeck labelings for binary lattice partitions, but provides only a "sketch of proof" with the critical inductive step—the existence of a vector $g_k$ of order 2 modulo $\Lambda_{k+1}$—deferred to Part II. Similarly, the properties of Barnes-Wall lattices and their principal sublattices in Section III are stated with proofs referenced to the forthcoming Part II. Since these decomposition results are foundational for the coset code constructions, distance calculations, and classification scheme throughout Part I, their omission makes the paper incomplete as a standalone rigorous treatment. It would be helpful to either include complete proofs of Lemma 2 and the Barnes-Wall lattice properties in Part I, or to clearly mark these results as provisional definitions pending rigorous proof in the sequel, particularly given the paper's stated intention that Parts I and II may be read independently.

**Unverified Claims of Geometric Uniformity for Nonlinear Trellis Codes**

In Section IV-B, the paper asserts that "All codes in this paper are distance-invariant" (geometrically uniform), which is essential for the error coefficient calculations in Tables IV-XI. However, for the nonlinear trellis codes presented in Classes I-VIII (Section VI), particularly those based on rate-$k/(k+1)$ encoders with potentially nonlinear labeling maps, the paper only establishes that regular labelings exist in principle (Section IV-C) but does not verify that the specific convolutional code constructions used in each class actually achieve this regularity. Since distance-invariance determines whether the error coefficient $N_0$ is well-defined and whether the minimum distance calculations are valid for performance comparisons, this gap undermines the coding gain claims. It would be helpful to provide either a general proof that the specific encoder structures in Classes I-VIII preserve distance-invariance or to qualify the error coefficient results as upper bounds for codes that may not be geometrically uniform.

**Overlapping and Non-Exhaustive Nature of the Geometrical Classification**

The paper presents Classes I-VIII in Section VI as a systematic categorization based on three binary characteristics, yet Table XI reveals significant overlap between classes (e.g., the four-state two-dimensional Ungerboeck code appears in both Class IV and Class VIII, and the two-state $Z^4/RZ^4$ code appears in Class I and Class V). Furthermore, the paper acknowledges in Section VII that some high-performance codes, such as Wei's 16-state four-dimensional code, do not fit neatly into this classification scheme. This suggests the eight classes are neither mutually exclusive nor exhaustive, limiting their utility as a systematic taxonomy for coset codes. It would be helpful to clarify in Section VI that these classes represent illustrative constructions rather than a partition of the code space, or to reorganize the classification to eliminate overlaps and explicitly address which known codes fall outside the scheme.

**Reliance on Infinite Lattice Approximations Without Finite Constellation Analysis**

The paper explicitly dismisses total coding gain $\gamma_{\mathrm{tot}}$ (which includes shaping gain for finite constellations) as "peripheral" and "confusing" in Section I-B, focusing exclusively on fundamental coding gain $\gamma$ for code comparisons throughout Tables IV-XI. However, this assumes that infinite lattice approximations hold for practical finite constellation sizes, and the paper does not quantify the approximation error or the shaping loss for the specific codes presented. For low-dimensional codes (e.g., two-dimensional Ungerboeck codes), the constellation boundary effects can significantly impact actual performance, making the fundamental coding gain potentially misleading for practical implementation where constellation expansion factors matter. It would be helpful to include in Section VII a quantitative analysis of the shaping loss for representative finite constellations, or at minimum to qualify the performance comparisons by noting that they apply asymptotically as constellation size increases.

**Unjustified Rule of Thumb for Effective Coding Gain Calculations**

In Section V, the paper introduces a "rule of thumb" that every factor of two increase in the error coefficient $\tilde{N}_0$ reduces the effective coding gain by approximately 0.2 dB at error rates around $10^{-6}$, and uses this to compute $\gamma_{\mathrm{eff}}$ in Tables IV-XI. However, this approximation lacks theoretical justification for the specific trellis code classes presented, particularly for the high-dimensional codes where the error coefficient may include contributions from multiple distance terms and the Gaussian approximation may not hold at moderate SNRs. Additionally, Table XI leaves many error coefficient entries blank (e.g., for the Class I $\Lambda_{16}/R\Lambda_{16}$ code), making performance comparisons incomplete. It would be helpful to provide a derivation or citation for the 0.2 dB rule specific to trellis-coded modulation, and to complete the missing error coefficient entries or explain why they cannot be computed within the paper's framework.

**Lack of Rigor in Verification of Regular Labelings for Multi-Level Partitions**

In Section IV-C, the paper defines a labeling as regular if the minimum squared distance between points in two cosets is a function only of the mod-2 sum of their labels, and asserts that regular labelings exist for all partitions used in the paper. The paper cites three conditions under which labelings are regular (linearity, Ungerboeck with equality, Cartesian products), but for the multi-level partition chains used in Classes III, IV, VII, and VIII (e.g., $Z^2/RZ^2/2Z^2$), the paper does not verify that the Ungerboeck distance bound actually holds with equality for all pairs of cosets at all levels, which is necessary for the labeling to be regular. Without this verification, the claim that all codes are distance-invariant rests on an unproven assertion about the existence of regular labelings for specific multi-level partitions. It would be helpful to provide explicit verification that regular labelings exist for the specific partition chains used in Classes III-VIII, or to restrict the distance-invariance claim to codes where regularity has been established.

**Status**: [Pending]

---

## Detailed Comments (16)

### 1. Definitional inconsistency between binary encoder restriction and ternary extensions

**Status**: [Pending]

**Quote**:
> A coset code is defined by a lattice partition $\Lambda / \Lambda'$ and by a binary encoder $C$ that selects a sequence of cosets of the lattice $\Lambda'$.

**Feedback**:
The abstract restricts coset codes to binary encoders, yet Section VII discusses ternary codes as coset codes. Readers might note that this creates a definitional inconsistency, as the restriction to binary encoders appears to exclude the ternary constructions discussed later. It would be helpful to generalize the definition to "encoder over a finite alphabet" or to explicitly scope the definition to binary lattices while noting extensions to larger alphabets.

---

### 2. Characterization of error coefficient and decoding complexity as geometric parameters

**Status**: [Pending]

**Quote**:
> The fundamental coding gain of a coset code, as well as other important parameters such as the error coefficient, the decoding complexity, and the constellation expansion factor, are purely geometric parameters determined by $C$ and $\Lambda / \Lambda'$.

**Feedback**:
Readers might note that the error coefficient depends on the number of minimum-weight codewords in the specific encoder $C$, which reflects combinatorial properties of the code structure beyond the geometry of the lattice partition. Similarly, decoding complexity depends on the trellis state count and branch structure. It would be helpful to characterize these as "structural parameters" rather than "purely geometric" to avoid ambiguity.

---

### 3. Unjustified restriction on Hamming distance equivalence

**Status**: [Pending]

**Quote**:
> as long as $d_H \leq 4$

**Feedback**:
The text asserts that for partitions $\mathbb{Z}^N/2\mathbb{Z}^N$, the minimum squared distance $d_{\min}^2$ equals the Hamming distance $d_H$ only when $d_H \leq 4$. Readers might note that the geometric argument supporting this restriction is not provided in the text, leaving the basis for this specific threshold unverified. It would be helpful to include the derivation showing why this equivalence requires $d_H \leq 4$ or to clarify the conditions under which the equality holds.

---

### 4. Missing applicability condition in Lemma 1 Corollary

**Status**: [Pending]

**Quote**:
> / \Lambda'] + \mathbb{R}(\Lambda)$.
> 
> **Corollary:** If $\Lambda$ is an integer lattice, then $V(\Lambda) = |\mathbb{Z}^N / \Lambda|$.
> 
> Not

**Feedback**:
Readers might note that this corollary requires $\Lambda$ to be a sublattice of $\mathbb{Z}^N$ for the index $|\mathbb{Z}^N/\Lambda|$ to be well-defined. For an integer lattice not contained in $\mathbb{Z}^N$, the quotient may be undefined or infinite. It would be helpful to specify "If $\Lambda$ is a sublattice of $\mathbb{Z}^N$" to ensure the quotient is well-defined.

---

### 5. Undefined terminology for mod-2 lattices

**Status**: [Pending]

**Quote**:
> $ code. (Unless otherwise specified, an $(N,K)$ code will always mean a linear binary $(N,K)$ code.)
> 
> **Lemma 3**: An $N$-dimensional real lattice $\Lambda$ is a mod-2 binary lattice if and only if it is the set of all integer $N$-tuples that are congruent modulo 2 to one of the codewords $c$ in a linear binary $(N, K)$ block code $C$. The redu

**Feedback**:
The term "mod-2 binary lattice" appears without explicit definition and seems inconsistent with the "2-depth $m$" notation introduced in Section II-E. Lemma 3 describes lattices of the form $2\mathbb{Z}^N + C$, which have 2-depth $m=1$, but the relationship between "mod-2" and "2-depth" is not clarified. It would be helpful to define "mod-2" explicitly as 2-depth equal to 1, or to reconcile these terminologies.

---

### 6. Description of matrix R as rotation operator

**Status**: [Pending]

**Quote**:
> The most important scaled orthogonal transformation for our purposes is the rotation operator $R$, defined by the $2 \times 2$ matrix
> 
> $$
> R \triangleq \left( \begin{array}{cc} 1 & 1 \\ 1 & -1 \end{array} \right).
> $$

**Feedback**:
Readers might note that the matrix $R$ has determinant $-2$, indicating it includes a reflection (improper rotation) rather than representing a pure rotation. Furthermore, no scaled rotation operator satisfies $R^2 = 2I$ while rotating by $45^\circ$. It would be helpful to describe $R$ as a "scaled orthogonal transformation" rather than a "rotation operator" to maintain mathematical precision.

---

### 7. Generalization in lattice designation D_k

**Status**: [Pending]

**Quote**:
> For example, the depth of $D_{k}$ is 1,

**Feedback**:
Readers might note that the properties listed (redundancy 1, informativity 1, normalized values $1/2$) hold specifically for $D_4$ (where $N=4$ real dimensions), not for general $D_k$. For $D_2$ ($N=2$), informativity is 0, and for $D_6$ ($N=6$), it is 2. It would be helpful to replace $D_k$ with $D_4$ to match the specific parameters discussed.

---

### 8. Approximate equality in Lemma 2 terminal condition

**Status**: [Pending]

**Quote**:
> $\Lambda_0 = \Lambda$, $\Lambda_1, \dots, \Lambda_K \approx \Lambda'$ such that

**Feedback**:
Readers might note that the lemma statement uses $\approx$ (isomorphic) for the terminal lattice, but the proof requires $\Lambda_K = \Lambda'$ to establish the partition chain ending at the specified sublattice. The relation $\approx$ is insufficient because the chain must terminate at the specific sublattice $\Lambda'$, not merely an isomorphic copy. It would be helpful to replace $\approx$ with $=$ to ensure the lemma applies to the intended sublattice.

---

### 9. Inconsistent modulus notation in complex lattice formulas

**Status**: [Pending]

**Quote**:
> If $\Lambda$ is a decomposable $N$-dimensional complex binary lattice of depth $\mu$, with code formula
> 
> $$
> \Lambda = \phi^ {n} \boldsymbol {G} ^ {N} + \phi^ {\mu - 1} C _ {\mu - 1} + \dots + C _ {0},
> $$

**Feedback**:
Readers might note that the text inconsistently uses $\phi^n$ and $\phi^u$ where the depth parameter $\mu$ is intended. The modulus for orthogonality is defined as $\phi^\mu$, and the finest lattice in the partition chain is $\phi^\mu G^N$. It would be helpful to replace all instances of $\phi^n$ and $\phi^u$ with $\phi^\mu$ to maintain consistency with the defined depth.

---

### 10. Inconsistent depth assignments for scaled Barnes-Wall lattices

**Status**: [Pending]

**Quote**:
> we regard $\phi^j\Lambda(r,n) \simeq R^j\Lambda(r,n)$ as having depth $n-r$ for any $j \geq 0$

**Feedback**:
Readers might note that multiplication by $\phi$ increases depth by 1, so $\phi^j\Lambda(r,n)$ should have depth $n-r+j$, not $n-r$. The diagram caption claims depth $n-r$ regardless of $j$, but the text assigns depth $n$ to lattices in the chain $\Lambda(0,n)/\phi\Lambda(1,n)/\cdots$. It would be helpful to clarify that depth increases with $j$ or to correct the depth assignment formula.

---

### 11. Normalized redundancy value in Table III

**Status**: [Pending]

**Quote**:
> | 3 | 1 | 1 | 4 | 8 | 60 | 3.75  |
> |  D8 | RE8 | 8 | 32 | 3 | 1 | 3/4 | 2 | 8 | 88 | 2.75  |
> |

**Feedback**:
Readers might note that the entry lists $\rho = 3/4$ for the partition $D_8/RE_8$. However, $\rho(\Lambda/\Lambda') = \rho(\Lambda) = r(\Lambda)/(N/2) = 1/4$ for $\Lambda = D_8$ (where $r=1, N=8$). The value $3/4$ appears inconsistent with the definition $\rho = r/(N/2)$. It would be helpful to verify and correct $\rho$ to $1/4$.

---

### 12. Duplicate partition entry with inconsistent parameters

**Status**: [Pending]

**Quote**:
>  | 4 | 8 | 60 | 3.75  |
> |  D8 | RE8 | 8 | 32 | 3 | 1 | 3/4 | 2 | 8 | 88 | 2.75  |
> |  D8 | RE8 | 8 | 128 | 3 | 1 | 1/4 | 2 | 8 | 280 | 2.2  |
> |  Z8 | RE8 | 8 | 256 | 3 | 1 | 0 | 1 | 8 | 504 | ~

**Feedback**:
Readers might note that Table III contains two rows for $D_8/RE_8$ with different orders (32 and 128) and decoding complexities. While $RE_8$ could denote different scalings, the standard definition gives $|D_8/RE_8| = 32$. The second entry appears to represent a different partition (e.g., $D_8/R^2E_8$) but uses the same label. It would be helpful to distinguish these with unique notation or to remove the duplicate entry.

---

### 13. State count formula uses redundancy instead of memory

**Status**: [Pending]

**Quote**:
> A trellis diagram for a $2^{r}$-state, rate-$k / (k + r)$ convolutional code is an exten

**Feedback**:
Readers might note that the number of states is determined by the constraint length $\nu$ (memory), not the redundancy $r$. For example, the four-state Ungerboeck code has $r=1$ but $\nu=2$ memory elements, giving $2^\nu = 4$ states, not $2^r = 2$. It would be helpful to replace $2^r$ with $2^\nu$ to reflect the standard relationship between memory and state count.

---

### 14. Algebraic inconsistency in volume equality chain

**Status**: [Pending]

**Quote**:
> Since $V(\mathbb{C}) = V(\Lambda_0) = 2^{-k(C)}V(\Lambda') = 2^{k+r}V(\Lambda') = 2^{r(C)}V(\Lambda)$

**Feedback**:
Readers might note that the equality $2^{-k(C)}V(\Lambda') = 2^{k+r}V(\Lambda')$ implies $2^{-k(C)} = 2^{k+r}$, which cannot hold for positive $k$ and $r$. From Lemma 1, $V(\Lambda') = 2^{k+r}V(\Lambda)$, so $V(\Lambda_0) = 2^{-k(C)} \cdot 2^{k+r}V(\Lambda) = 2^r V(\Lambda) = 2^{r(C)}V(\Lambda)$. It would be helpful to correct the intermediate term to $2^{r(C)}V(\Lambda)$.

---

### 15. Exponent in coding gain formula

**Status**: [Pending]

**Quote**:
> The fundamental coding gain $\gamma$ is given by the formula $2^{-e}d_{\mathrm{min}}^2$

**Feedback**:
Readers might note that the formula uses $e$ (where $2^e$ is the number of states) rather than the normalized redundancy $\rho$. For the one-dimensional codes with $\rho=2$ and $d_{\min}^2=10$, the formula gives $10/2^3 = 1.25$ (for $e=3$), but the table lists $\gamma = 2.5 = 10/2^2 = 10/2^\rho$. It would be helpful to replace $2^{-e}$ with $2^{-\rho}$ to match the tabulated values.

---

### 16. Branch count formula variable substitution

**Status**: [Pending]

**Quote**:
> where $\beta = 2 - 2^{-k}$, and $2^{k + r}$ is the number of branches per stage of the trellis,

**Feedback**:
Readers might note that the number of branches per stage equals $2^{k+e}$ (where $2^e$ is the number of states), not $2^{k+r}$. For the four-state Ungerboeck code ($e=2, r=1$), there are $2^{k+e} = 8$ branches, not $2^{k+r} = 4$. It would be helpful to replace $r$ with $e$ (or $\nu$) to correctly represent the branch count.

---
