# VertiPaq Binary Format Specification

Independent documentation of the Power BI VertiPaq columnar storage format, derived through clean-room analysis of file structures for interoperability purposes.

## PBIX File Structure

A `.pbix` file is a standard ZIP archive containing:

| Entry | Format | Description |
|-------|--------|-------------|
| `DataModel` | XPress9 compressed | ABF archive containing all VertiPaq data |
| `Report/Layout` | JSON | Report pages, visuals, filters, bindings |
| `Settings` | JSON | Report settings |
| `Metadata` | JSON | Package metadata |
| `SecurityBindings` | Binary | Encrypted credential references |
| `[Content_Types].xml` | XML | OPC package manifest |

## ABF Archive

The DataModel entry, after XPress9 decompression, is an ABF (Analysis Backup Format) archive:

```
ABF Structure:
  BackupLog.xml    — File manifest with offsets and sizes
  VirtualDirectory — XML index of all contained files
  metadata.sqlitedb — SQLite database with all metadata
  *.idf            — Column data (bit-packed + RLE)
  *.idfmeta        — Column segment statistics
  *.dictionary     — Column dictionary encoding
  *.hidx           — Hash index files
```

### BackupLog XML

Contains `<FileGroup>` and `<File>` elements with absolute byte offsets into the ABF binary:

```xml
<FileGroup Name="metadata.sqlitedb" Offset="1234" Size="315392" EndOfFile="315392" />
```

### VirtualDirectory XML

Maps logical paths to file entries, establishing the folder hierarchy:

```xml
<File Name="tablename (id).tbl\partition.prt\0.colname.0.idf" ... />
```

## SQLite Metadata Schema

The `metadata.sqlitedb` contains the complete tabular model definition:

### Core Tables

| Table | Purpose | Key Fields |
|-------|---------|------------|
| `[Table]` | User and system tables | ID, Name, TableStorageID, SystemFlags |
| `[Column]` | Column definitions | ID, TableID, Type, DataType, ColumnStorageID |
| `[Partition]` | Data partitions | ID, TableID, Type, Mode, QueryDefinition |
| `[Relationship]` | Table relationships | ID, FromTableID, FromColumnID, ToTableID, ToColumnID |
| `[Measure]` | DAX measures | ID, TableID, Name, Expression |
| `[AttributeHierarchy]` | Column hierarchies | ID, ColumnID |

### Storage Tables

| Table | Purpose |
|-------|---------|
| `TableStorage` | Links Table → StorageFolder |
| `PartitionStorage` | Links Partition → SegmentMapStorage + StorageFolder |
| `ColumnStorage` | Column statistics and dictionary reference |
| `DictionaryStorage` | Dictionary type, BaseId, LastId, IsOperatingOn32 |
| `ColumnPartitionStorage` | Links column to IDF/IDFMETA files |
| `SegmentStorage` | Segment count and IDFMETA file reference |
| `SegmentMapStorage` | RecordCount, SegmentCount, RecordsPerSegment |
| `StorageFolder` | ABF path for table/partition directories |
| `StorageFile` | ABF path for individual data files |
| `AttributeHierarchyStorage` | MaterializationType, SystemTableID, statistics |

### Partition Types and Modes

| Partition.Type | Meaning |
|---------------|---------|
| 3 | System partition (H$, R$ tables) |
| 4 | M/Power Query partition (Import and DirectQuery) |
| 6 | PolicyRange (incremental refresh) |

| Partition.Mode | Meaning |
|---------------|---------|
| 0 | Import (data stored in VertiPaq) |
| 1 | DirectQuery (live database queries) |
| 2 | System (H$, R$ tables) |

## IDF Format (Column Data)

### XMHybridRLECompressionInfo (u32_a = 0xABA5A)

Used for regular data columns. Hybrid RLE + bit-packed encoding:

```
Primary Segment:
  u64: primary_entry_count (always 16 slots)
  16 × (u32 data_value, u32 repeat_count):
    - data_value = 0xFFFFFFFF → next repeat_count values in sub-segment
    - Otherwise → repeat data_value exactly repeat_count times

Sub Segment:
  u64: sub_segment_size (in u64 words)
  N × u64: bit-packed values at specified bit_width
```

#### Bit Width Computation

**Critical**: bit_width must be computed from the number of distinct values, NOT from `max_data_id + 1`:

```
distinct = number of unique non-null values in the column
if distinct <= 2:
    bit_width_raw = 1
else:
    bit_width_raw = ceil(log2(distinct))
bit_width = align_to_valid_N(max(1, bit_width_raw))
```

The IDFMETA u32_b field must use the same aligned bit_width. A mismatch between the IDF encoding bit width and IDFMETA u32_b causes `QuerySystemError` crashes in PBI Desktop.

#### Dictionary Order

The dictionary determines the mapping between values and data_ids:

- **String columns**: insertion order (order values first appear in the row data)
- **Numeric columns** (Int64, Double, DateTime, Decimal, Boolean): sorted ascending

The first dictionary entry gets data_id = DATA_ID_OFFSET (3). All dependent structures (H$ POS_TO_ID, IDF data_ids) must use the same ordering.

### XMRENoSplitCompressionInfo<N> (u32_a = 0xABA36 + N)

Used for H$ and R$ system table columns. Raw N-bit values:

```
  u64: header (bit width indicator)
  Segments: N-bit values packed into u64 words
  Padding to 8-byte alignment between segments
```

Valid N values: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 16, 21, 32

### XM123CompressionInfo (u32_b = 0xABA5B)

Used for RowNumber columns. Sequential integer encoding.

## IDFMETA Format (Segment Statistics)

Tagged block structure with XML-like open/close tags:

```
<1:CP\0  (6 bytes) — Column Partition open
  u64: version (1 for single segment, 2+ for multi-segment)

  <1:CS\0  (6 bytes) — Column Segment open
    u64: record_count
    u64: one (1 for data columns, 0 for system/RowNumber)
    u32: u32_a (compression family selector)
    u32: u32_b (compression class selector)
    ... compression-specific fields ...

    <1:SS\0  (6 bytes) — Segment Statistics open
      u64: distinct_states
      u32: min_data_id
      u32: max_data_id
      u32: original_min_segment_data_id
      i64: rle_sort_order (-1 = unsorted)
      u64: row_count
      u8:  has_nulls
      u64: rle_runs
      u64: others_rle_runs
    SS:1>\0  (6 bytes) — close

    u8: has_bit_packed_sub_segment
    [optional sub-CS block if has_bit_packed=1]

  CS:1>\0  (6 bytes) — close
  [repeat CS blocks for multi-segment columns]

CP:1>\0  (6 bytes) — close

<1:SDOs\0  — Segment Data Offsets
  <1:CSDOs\0  — per-segment byte offsets into IDF
    u64: sub_segment_size (byte offset)
    u64: count
  CSDOs:1>\0
  [repeat per segment]
SDOs:1>\0
```

### IDFMETA for Data Columns

For standard data columns (XMHybridRLE):
- `u32_b` = `0xABA36 + aligned_bit_width` — MUST match the IDF encoding bit width
- `distinct_states` = number of distinct values in the column
- `min_data_id` = DATA_ID_OFFSET (3) for non-empty columns
- `max_data_id` = DATA_ID_OFFSET + distinct - 1
- `original_min_segment_data_id` = 2 (PBI Desktop convention)
- `row_count` = total number of rows

### IDFMETA for System Columns (H$, R$)

For NoSplit-encoded system columns:
- `one` = 0 (system flag)
- `distinct_states` = 0
- `min_data_id` = 2
- `max_data_id` = 2
- `original_min_segment_data_id` = 2
- `row_count` = 0

## Compression Class ID Selectors

Determined through binary format analysis:

```
u32_a (compression family):
  0xABA5A → XMHybridRLECompressionInfo (standard data columns)
  0xABA36 + N → XMRENoSplitCompressionInfo<N> (system columns)

u32_b (inner compression class):
  0xABA36 + aligned_bw → XMRENoSplitCompressionInfo<aligned_bw>
  0xABA5B → XM123CompressionInfo (RowNumber)
  0xABA57 → XMREGeneralCompressionInfo

aligned_bw = bit width rounded to valid N: {1,2,3,4,5,6,7,8,9,10,12,16,21,32}
```

## Dictionary Format

### Integer Dictionary (dict_type = 0, LONG)

```
u32: dict_type (0)
u32: entry_count
u32[]: values (4 bytes each if IsOperatingOn32=1, 8 bytes if =0)
```

Values are sorted ascending for numeric types.

### Float Dictionary (dict_type = 1, REAL)

```
u32: dict_type (1)
u32: entry_count
f64[]: values (8 bytes each, IEEE 754)
```

Values are sorted ascending.

### String Dictionary (dict_type = 2, STRING)

```
u32: dict_type (2)
u8: f_store_compressed (1 = compressed)
u32: store_longest_string (max string length excluding null)
u32: char_count (total UTF-16LE characters)
u16[]: char_data (UTF-16LE encoded, null-separated)
u32: record_count
RecordHandle[]: handles
  u32: char_offset (into char_data)
  u32: page_id (always 0)
HashInfo:
  u32: hash_table_size
  u32[]: hash_entries (chain pointers, -1 = empty)
  u32[]: chain_next (-1 = end of chain)
```

String dictionary entries use **insertion order** (order values first appear in the source data), not sorted order. This is critical for correct H$ hierarchy alignment.

### Compressed string pages (canonical Huffman)

When a page's uncompressed character store grows large (Power BI Desktop
switches above ~16 KB of UTF-16 text), the page is stored Huffman-compressed
instead. The format is documented publicly in Microsoft's open specification
[MS-XLDM §2.7.4 — Huffman Compression](https://learn.microsoft.com/en-us/openspecs/office_file_formats/ms-xldm/f70b41f2-ca64-44a1-9e6f-53e63f6a5ee9).

```
CompressedStrings:
  u32:  store_total_bits          (end-of-stream sentinel; last string's end)
  u32:  character_set_type_id     (0x000ABA91 single, 0x000ABA92 general)
  u64:  len_compressed_buffer
  u8:   character_set_used        (only when type == 0x000ABA91)
  u32:  ui_decode_bits            (primary-table width hint, ≤ 12)
  u8[128]: encode_array           (256 nibble-packed code lengths, 2–15 bits)
  u64:  buffer_size               (== len_compressed_buffer)
  u8[]: compressed_string_buffer  (bitstream; adjacent bytes pair-swapped)
```

- **encode_array**: 128 bytes; low nibble of byte *i* is the code length for
  symbol `2i`, high nibble for `2i+1`. Length 0 = symbol unused.
- **Canonical codes**: sort `(length, symbol)` ascending; start at code 0 and
  increment, left-shifting on each length increase (MS-XLDM §2.7.4.1.5).
- **Bit order**: codes are packed MSB-first (big-endian); the resulting byte
  buffer then has every adjacent pair swapped (`2k ↔ 2k+1`) on disk. Buffer
  length is `ceil(total_bits/8)` rounded up to even, plus 2.
- **Charset modes**: *single* (`0x000ABA91`) Huffman-encodes only the low byte
  of each UTF-16 character and stores the common high byte once as
  `character_set_used` (used for Latin text); *general* (`0x000ABA92`) encodes
  every byte of the UTF-16LE stream.
- **Record handles**: for compressed pages, the record-handle offset is the
  string's **start bit** (not char offset). Pages hold at most 2^19 UTF-16
  characters; larger dictionaries split across multiple compressed pages that
  share one trailing record-handle vector.

This layer delegates the canonical-Huffman codec to the MIT
[`xmhuffman`](https://github.com/Hugoberry/xmhuffman-cython) primitive, the
same way the ZIP layer delegates XPress9 to `xpress9-python`.

## Zero-Row Tables

A table with columns but no rows is a valid VertiPaq state — Power BI Desktop
produces one itself (the conventional `# Measures` holder table). Its
representation differs from a populated table in three specific ways:

| Structure | Zero-row value | Populated value |
|-----------|---------------|-----------------|
| `SegmentMapStorage` | RecordCount=0, SegmentCount=1, RecordsPerSegment=0 | RecordCount=rows, RecordsPerSegment=rows |
| `AttributeHierarchyStorage.MaterializationType` | 2 (DistinctDataCount=0), no H$ table | 0 for user columns (H$ materialized); 3 for RowNumber |
| String dictionary | **no page** (`store_page_count=0`) | one or more pages |

The partition itself is unchanged: Type=4 (M/Power Query), Mode=0 (Import),
State=1, DataView=3.

**The string store is the critical case.** Emitting a page with
`allocation_size = 0` produces a zero-size (NULL) character buffer, which the
engine's string-store consistency check rejects — the file then fails to open
with `PFE_XM_DBCC_STRINGSTORE_CORRUPT` and Desktop silently falls back to
creating an empty database. A store containing no strings must carry **no page
at all**. Numeric dictionaries are unaffected: an empty numeric dictionary is
simply a `VectorOfVectors` with count 0 and has no page/allocation concept.

## H$ System Tables (Attribute Hierarchies)

Each user column with MaterializationType=0 has an H$ system table:

```
Table name: H$<table> (<tid>)$<column> (<cid>)
Columns: POS_TO_ID (sorted position → data_id)
          ID_TO_POS (data_id → sorted position)
Encoding: NoSplit<32> (u32_a = 0xABA56)
SMS: RecordCount = distinct + 3, SegmentCount = ceil(RecordCount/RecPerSeg)
ColumnStorage: distinct=1, min=2, max=2, orig_min=2, rows=0
```

**Critical**: POS_TO_ID and ID_TO_POS must be computed using the same dictionary ordering as the VertiPaq encoder. For String columns (insertion order), the "sorted position" refers to the lexicographic sort of the insertion-ordered dictionary. For numeric columns (already sorted), POS_TO_ID is an identity mapping offset by DATA_ID_OFFSET.

## R$ System Tables (Relationships)

Each relationship has an R$ system table:

```
Table name: R$<from_table> (<tid>)$<guid> (<rid>)
Columns: INDEX (from-table FK data_id → to-table row mapping)
Encoding: NoSplit<N> where N = bits_needed(max_row_value)
RecordCount: distinct_FK_values + DATA_ID_OFFSET (3)
Layout:
  positions 0-2: padding zeros (DATA_ID_OFFSET)
  position 3+i:  1-based row index into TO table for FK dict entry i
Values: 1-based row indices (RowNumber-style, NOT 0-based)
IDFMETA: one=0, min_data_id=0
SMS: RecordCount = distinct + 3, SegmentCount = 1
ColumnStorage: distinct=1, min=2, max=2, orig_min=2, rows=0
DictionaryStorage: Type=0, DataType=19, BaseId=0 (no external dictionary)
```

The R$ INDEX is addressed by the FK column's data_id. Since data columns use
DATA_ID_OFFSET=3, position 3 corresponds to the first dictionary entry. The
stored value is the 1-based row index into the TO table (matching RowNumber
values, which start at 1).

### R$ Verification

Verified through binary comparison against PBI Desktop ground truth:

| Property | Generated | Ground Truth | Match |
|----------|-----------|-------------|-------|
| RecordCount (6 distinct FK) | 9 | 9 | ✅ |
| RecordCount (3 distinct FK) | 6 | 6 | ✅ |
| Padding positions 0-2 | [0, 0, 0] | [0, 0, 0] | ✅ |
| Data positions 3+ | 1-based indices | 1-based indices | ✅ |
| ColumnStorage stats | D=1,min=2,max=2 | D=1,min=2,max=2 | ✅ |
| DictionaryStorage | Type=0,DT=19,Base=0 | Type=0,DT=19,Base=0 | ✅ |

## XPress9 Compression

The DataModel entry uses Microsoft XPress9 block compression:

- Block size: typically 65536 bytes
- Header: 4-byte magic + metadata
- Byte-exact round-trip verified

## Key Constants

| Constant | Value | Meaning |
|----------|-------|---------|
| DATA_ID_OFFSET | 3 | First dictionary entry gets data_id = 3 |
| BaseId | 2 | DictionaryStorage.BaseId for data columns |
| HYBRID_RLE_FAMILY | 0xABA5A | XMHybridRLECompressionInfo |
| NOSPLIT_BASE | 0xABA36 | XMRENoSplitCompressionInfo base |
| XM123_CLASS | 0xABA5B | XM123CompressionInfo (RowNumber) |

---

*This specification was independently derived through analysis of file structures for interoperability purposes. Reverse engineering undertaken to achieve interoperability is recognized as a permitted purpose under, among others, the EU Software Directive (2009/24/EC, Art. 6) and the US DMCA (§1201(f)), subject to the conditions those provisions set out; this note is provided for context and is not legal advice (see [legal-and-cleanroom.md](legal-and-cleanroom.md)). No Microsoft source code was used.*
