# Architecture

## Overview

pbix-mcp is an MCP (Model Context Protocol) server that provides programmatic access to Power BI `.pbix` and `.pbit` files. It exposes 133 tools via stdio transport. Every tool is also a plain importable Python function, so the whole API is usable in-process without an MCP client (see [html-visuals.md](html-visuals.md#pure-python-usage)).

## Module Layout

```
src/pbix_mcp/
  server.py              # MCP tool definitions (133 tools)
  cli.py                 # Entry point with --log-level flag
  builder.py             # PBIX creation (entirely from scratch — metadata, VertiPaq, ABF, layout)
  builder_v2.py          # Template-free ABF + ZIP generation
  html_templates.py      # HTML/SVG template builders (kpi_card, bar_chart, gauge, table, …)
  svg_measures.py        # DAX codegen for SVG data-URI image measures (data_bar, sparkline, …)
  report_binding.py      # Compile query + dataTransforms for visual containers
  errors.py              # Typed exception hierarchy
  logging_config.py      # Structured logging (normal/debug/trace)
  dax/
    engine.py            # DAX evaluator (verified parity: all 435 query-evaluable DAX functions; the other 32 are non-evaluable in Desktop itself)
    calc_tables.py       # Calculated table + column evaluation
  formats/
    abf_rebuild.py       # ABF archive read/write/build
    datamodel_roundtrip.py  # XPress9 compress/decompress
    vertipaq_encoder.py  # VertiPaq column encoding + NoSplit<N> encoder
  models/
    responses.py         # Pydantic response models (JSON output)
    requests.py          # Pydantic input models
```

## Data Flow

### Reading a PBIX

1. `pbix_open` extracts the ZIP to a temp directory (members are validated first — decompression-bomb size/ratio caps and path-traversal/symlink rejection)
2. Report/Layout is parsed as JSON for visual/page operations
3. DataModel is XPress9-compressed; decompress to get ABF archive
4. ABF contains SQLite metadata + VertiPaq column data
5. Native VertiPaq decoder (vertipaq_decoder.py + model_reader.py) reads materialized table data
6. calc_tables.py evaluates calculated tables/columns from metadata DAX
7. DAX engine evaluates measures against the loaded data

### Writing a PBIX

1. Modifications are applied to files in the temp directory (writes that incorporate caller-supplied names — e.g. a theme `filename`, `.pbiviz` members — are contained to the work dir via `_safe_join`, so they cannot escape via path traversal)
2. `pbix_save` repacks the temp directory into a ZIP
3. SecurityBindings are auto-removed to prevent corruption
4. DataModel is stored uncompressed (it's already XPress9)

### Creating a PBIX

**Everything is generated from scratch** — no templates or skeletons. The PBIX binary structures pbix-mcp's capabilities require are independently reimplemented: PBIX ZIP shell, ABF binary container (signature, header, VirtualDirectory, BackupLog), XMLA database document (db.xml), metadata SQLite (68 system tables), VertiPaq column storage, and report layout JSON. Every artifact is generated, including the 144-byte `CryptKey.bin` (a fixed-format container: observed scaffold + self-authored key region — see docs/reverse-engineering/experiments/cryptkey.md).

1. `PBIXBuilder` generates clean SQLite metadata (DATASOURCEVERSION=2) — only user-specified tables, columns, and measures
2. Key PBI annotations are written: PBI_IsFromSource (ObjectType=7), PBI_ResultType, SummarizationSetBy, PBI_QueryOrder, __PBI_TimeIntelligenceEnabled
3. Fixed RowNumber GUID (2662979B-1795-4F74-8F37-6A1BA8059B61) ensures stable attribute hierarchy references
4. VertiPaq encoder writes actual row data into column segments using pure bitpack (RLE disabled — slightly less space-efficient but correct). Verified with 6 tables, 36 columns, 5 relationships, 25 rows, 3 pages, 14 visuals (Northwind showcase)
5. H$ attribute hierarchy tables are generated with sorted POS_TO_ID/ID_TO_POS using NoSplit<32>
6. R$ relationship index tables use +3 DATA_ID_OFFSET padding and 1-based row indices (verified byte-exact against PBI Desktop ground truth)
7. Relationships auto-detect Many/One sides; From=Many (fact table), To=One (dimension table)
8. ABF binary container is built from scratch — signature, header, VDir, BackupLog, db.xml, CryptKey, all data files laid out sequentially
9. ABF is XPress9-compressed into a DataModel
10. Report/Layout JSON is generated from scratch with a default page and visuals (table, pieChart, clusteredBarChart, card, slicer all supported)
11. PBIX ZIP shell is generated from scratch (Version, Content_Types, DiagramLayout, Settings, Metadata); packaged as a valid PBIX
12. For database sources, M expressions use `Item` key (not `Name`) for MySQL/PostgreSQL table navigation

### Data Source Support

| Source | Import Mode | DirectQuery | Refresh Verified |
|--------|------------|-------------|------------------|
| Embedded data | Yes | N/A | N/A |
| CSV files | Yes | N/A | Yes |
| SQLite | Yes | N/A | Yes |
| SQL Server | Yes | Yes | Yes |
| MySQL | Yes | Yes (MariaDB ODBC 3.1) | Yes |
| PostgreSQL | Yes | Yes (native) | Yes |
| Excel | Yes | N/A | Yes |
| JSON/API | Yes | N/A | Yes |
| Azure SQL | Yes | Yes | Yes |

## VertiPaq Encoding Details

### IDF (Column Data)

Data columns use XMHybridRLECompressionInfo (u32_a = 0xABA5A). RLE is disabled; pure bitpack is used.

**Bit width computation:**
```
distinct = len(unique_values)
if distinct <= 2:
    bit_width_raw = 1
else:
    bit_width_raw = ceil(log2(distinct))
bit_width = align_to_valid_N(max(1, bit_width_raw))
```

Valid N: {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 16, 21, 32}

**Critical**: bit_width must be computed from `distinct_count` (not `max_data_id + 1`). The IDFMETA u32_b compression class selector must use the same aligned bit_width. A mismatch between IDF encoding and IDFMETA metadata causes `QuerySystemError` crashes.

### Dictionary Order

- **String columns**: insertion order (order values first appear in the row data)
- **Numeric columns** (Int64, Double, DateTime, Decimal, Boolean): sorted ascending

The H$ attribute hierarchy POS_TO_ID/ID_TO_POS must use the same sort order as the encoder's dictionary. Mismatched order causes hierarchy lookup failures.

### R$ Relationship Index Tables

R$ tables map FK column data_ids to TO table row indices:

```
RecordCount = distinct_FK_values + 3  (DATA_ID_OFFSET)

IDF layout (NoSplit<N>):
  position 0: 0  (padding)
  position 1: 0  (padding)
  position 2: 0  (padding)
  position 3: 1-based row index for FK dict entry 0
  position 4: 1-based row index for FK dict entry 1
  ...
  position 3+i: 1-based row index for FK dict entry i
```

The R$ is addressed by the FK column's data_id. Since data columns use DATA_ID_OFFSET=3, the first dictionary entry has data_id=3, and R$ position 3 maps it to the target row. Values are 1-based (matching RowNumber values, which start at 1).

### H$ Attribute Hierarchy Tables

Each user column with MaterializationType=0 has an H$ table with two columns:
- **POS_TO_ID**: sorted position → data_id mapping
- **ID_TO_POS**: data_id → sorted position mapping (inverse of POS_TO_ID)

Encoding: NoSplit<32> (u32_a = 0xABA56). RecordCount = distinct + 3.

## Verification Methodology

The VertiPaq binary format was verified through a ground truth comparison approach:

### Ground Truth Generation
1. A small test dataset was created: 3 tables (Orders, Products, Suppliers), 2 relationships, 6 rows each
2. The dataset was loaded into PBI Desktop and saved as `test_ground_truth_v2.pbix`
3. PBI Desktop performed a Refresh, which causes the engine to regenerate all VertiPaq binary data (IDF, IDFMETA, dictionaries, H$, R$)

### Binary Comparison
1. Both the generated PBIX and ground truth PBIX were decompressed (XPress9 → ABF)
2. The ABF archives were unpacked to extract individual files
3. The metadata SQLite was queried for SegmentMapStorage (RecordCount), ColumnStorage (statistics), DictionaryStorage (type/BaseId), and Relationship definitions
4. R$ IDF files were decoded using NoSplit<N> bit-unpacking at the bit width specified by IDFMETA u32_a
5. Values were compared position-by-position

### Key Discoveries

| Component | Initial assumption | Ground truth finding | Fix |
|-----------|-------------------|---------------------|-----|
| R$ RecordCount | = total FK rows | = distinct FK values + 3 | Changed SMS and IDF |
| R$ padding | None (0-indexed) | 3 zeros at positions 0-2 | Added DATA_ID_OFFSET prefix |
| R$ values | 0-based row indices | 1-based row indices | Added +1 to all values |
| IDF bit_width | ceil(log2(max_data_id+1)) | ceil(log2(distinct_count)) | Changed computation |
| IDFMETA u32_b | Independent of IDF | Must match IDF bit_width | Aligned both |
| Dict order (String) | Sorted | Insertion order | Changed to insertion |
| H$ sort order | Independent | Must match encoder dict | Aligned with encoder |

## Error Handling

Every tool catches `PBIXMCPError` (typed) before `Exception` (generic).
All responses are JSON via `ToolResponse.to_text()`:

```json
{"success": true, "message": "...", "data": ..., "warnings": [...]}
{"success": false, "error_code": "PBIX_INVALID", "message": "..."}
```

## DAX Engine

The engine has verified parity with Power BI Desktop on **100% of the DAX surface
Desktop can evaluate in a query** — not a strict Analysis Services runtime. It
implements every one of the 435 query-evaluable functions in the engine's
467-function catalog, pinned by per-function goldens (1e-9 tolerance) and
full-corpus 1:1 checks (see [dax-coverage.md](dax-coverage.md)). The other 32 of
the 467 are refused by Desktop itself in a query (visual-calc-only,
calculation-group-only, edition-gated, …), so there is nothing to match. Key
design decisions:

- Returns `None` for unsupported functions (tracked in `unsupported_functions`)
- Raises `DAXEvaluationError` for circular references (caught by graceful degradation → `None`)
- Uses heuristics for date-table detection (column named "Date" in a table)
- Auto-applies default slicer filters from report layout
- Smart SELECTEDVALUE fallback for parameter-driven measures
