# Known Limitations

## DAX Engine

The DAX engine has **verified parity with Power BI Desktop on 100% of the DAX
surface Desktop can evaluate in a query** — every one of the 435 query-evaluable
functions in the engine's 467-function catalog is implemented, pinned by
per-function goldens captured from Desktop's own engine (1e-9 tolerance) plus
full-corpus 1:1 checks (see [DAX Engine](../README.md#dax-engine) and
[dax-coverage.md](dax-coverage.md)). The other 32 of the 467 are **not a coverage
gap**: Desktop itself refuses them in a query (visual-calculation-only,
calculation-group-only, edition-gated, etc.), so parity means refusing them the
same way — they return `None` with status `"unsupported"`, never a guess. The
engine is not a strict Analysis Services runtime, so the bounds below still apply.

| Behavior | What happens | Impact |
|----------|-------------|--------|
| Unsupported function | Returns `None`, tracked in `unsupported_functions` | Measure result is `None` with status "unsupported" |
| Circular reference | Raises `DAXEvaluationError`, caught by graceful degradation | Measure result is `None`; no infinite loop |
| Date-table detection | Heuristic: looks for column named "Date" | May pick wrong table in ambiguous models |
| Bare-table iterators | `TOPN`/`RANKX` over a bare table can return BLANK (`SUMX`/`MAXX`/`AVERAGEX`/`MINX`/`COUNTX` handle it correctly as of 0.9.9) | Iterate `TOPN`/`RANKX` over `VALUES()`/`ALL(Table[Col])` where possible |
| Runaway measures | A measure is bounded to a fixed number of sub-expression evaluations; a non-terminating/expansion runaway degrades to BLANK rather than hanging | A single pathologically-slow (e.g. O(n²)) measure can still be slow but no longer hangs indefinitely on runaway expansion |
| Large tables | In-memory Python, no VertiPaq compression | Performance degrades at millions of rows |
| RANKX visual row context | Returns BLANK | 1 out of 204 tested measures affected |
| Running-total accumulation form | `SUMX(FILTER(ALL(D), D[K] <= _at), CALCULATE(...))` re-scans + context-transitions per iterated value — **O(n²)** in distinct values; measured not finishing in 7 min over ~1,850 values, while the single-pass form returns in ~1 s (issue #25 r23#4) | Write running totals as `CALCULATE(agg, FILTER(ALL(D[K]), D[K] <= MAX(D[K])))` — one filtered pass, no per-row CALCULATE |

## File Formats

| Format | Support |
|--------|---------|
| .pbix (Import mode) | Full read/write/create. Entire PBIX generated from scratch — no templates. Open, Refresh verified for all database types |
| .pbit (Template) | Full read/write |
| DirectQuery (create) | Open, live data, Refresh all verified — PostgreSQL (native), MySQL (MariaDB ODBC 3.1), SQL Server |
| DirectQuery (open existing) | Layout, measures, and metadata editing work; DAX eval and table reads unavailable (data lives in the remote source) |
| Composite models | Not tested |
| Live connections | Not supported |
| PBIR layout | Full read AND write — PBIR (`Report/definition`) files are edited in place (create/delete pages and visuals, filters, geometry) through the same tools as classic `Report/Layout` |

## HTML / CSS / SVG visuals

The bundled `PBIX HTML` custom visual (`pbix_add_html_visual`, see
[html-visuals.md](html-visuals.md)) has these constraints:

| Limitation | Detail |
|--------|---------|
| No external resources | Power BI's visual sandbox blocks all network requests — no `<script src>`, `<link>`, remote `<img>`/font, or `@import`. Inline everything; embed images as base64 `data:` URIs. |
| Content size | The HTML content measure must stay under ~32,000 characters (Analysis Services silently truncates a longer text cell); `pbix_add_html_visual` raises before the limit. |
| Layout formats | Both formats are supported (since 0.9.39): `pbix_add_html_visual` works on legacy `Report/Layout` AND PBIR `Report/definition` files — on PBIR the visual is written in place and `publicCustomVisuals` is synced into `report.json`. |
| Cross-filter needs a bound field | `category_field` cross-filtering only affects visuals reachable (through model relationships) from the bound column — same as any native visual. |
| Uncertified | The visual uses `innerHTML` and is intentionally uncertified (not published to AppSource). **Consequence: it renders EMPTY in the Power BI service** under default tenant trust policy (uncertified file-embedded visuals are blocked) — it is a Desktop-only capability. For service-portable rich content use [rich-content.md](rich-content.md) (SVG image measures, Deneb references). |

## Data Sources (from-scratch creation)

| Source | Import Mode | DirectQuery | Notes |
|--------|------------|-------------|-------|
| Embedded data | Open ✅, Refresh N/A | N/A | Default — data in PBIX, no external source |
| CSV files | Open ✅, Refresh ✅ | N/A | `source_csv` — Refresh re-imports from CSV |
| SQLite | Open ✅, Refresh ✅ | N/A | `source_db` — requires SQLite3 ODBC driver |
| SQL Server | Open ✅, Refresh ✅ | Open ✅, Live ✅, Refresh ✅ | `source_db` — verified with LocalDB |
| MySQL | Open ✅, Refresh ✅ | Open ✅, Live ✅, Refresh ✅ | DirectQuery requires MariaDB ODBC 3.1 driver (`type: 'mariadb'`) |
| PostgreSQL | Open ✅, Refresh ✅ | Open ✅, Live ✅, Refresh ✅ | `source_db` — verified with PostgreSQL 16 (native DirectQuery) |
| Excel | Open ✅, Refresh ✅ | N/A | `source_db` with `type: 'excel'` |
| JSON/API | Open ✅, Refresh ✅ | N/A | `source_db` with `type: 'json'` |
| Azure SQL | Open ✅, Refresh ✅ | Open ✅, Live ✅, Refresh ✅ | `source_db` with `type: 'azuresql'` |

## Supported Data Types

| Type | Status | Dictionary Format |
|------|--------|-------------------|
| String | ✅ Stable | UTF-16LE with hash table; large dictionaries canonical-Huffman-compressed (MS-XLDM §2.7.4, read + write) |
| Int64 | ✅ Stable | External 32-bit entries (IsOperatingOn32=1) |
| Double | ✅ Stable | External 64-bit IEEE 754 entries |
| DateTime | ✅ Stable | External 64-bit entries (same encoding as Double) |
| Decimal | ✅ Stable | External 32-bit entries (value × 10000, IsOperatingOn32=1) |
| Boolean | ✅ Stable | External 32-bit entries (0/1, IsOperatingOn32=1) |

## Builder

### What is generated from scratch
Every layer of the PBIX is generated from scratch — no templates or skeletons:

- **PBIX ZIP shell**: Version, Content_Types, DiagramLayout, Settings, Metadata — all generated constants
- **ABF binary container**: signature, BackupLogHeader, VirtualDirectory, BackupLog — `build_abf_clean()`
- **XMLA Load document (db.xml)**: 28 xmlns namespaces, CompatibilityLevel=1550 — `generate_db_xml()`
- **CryptKey.bin**: 144-byte fixed-format container, independently generated (observed scaffold + self-authored key region; database-independent) — see docs/reverse-engineering/experiments/cryptkey.md
- **Metadata SQLite**: clean DATASOURCEVERSION=2, 68 system tables — only user-specified tables, columns, and measures
- **VertiPaq column data**: all IDF segments, dictionaries, H$ hierarchy tables, R$ relationship tables. Verified with 6 tables, 36 columns, 5 relationships, 25 rows, 3 pages, 14 visuals (Northwind showcase). Cross-table lookups verified byte-exact against PBI Desktop ground truth.
- **Report layout JSON**: pages, visuals, filters generated from scratch. Supported visuals: table, pieChart, clusteredBarChart, clusteredColumnChart, card, slicer
- **XPress9 compression**: custom compress/decompress with reversed chunk framing and headers

### VertiPaq encoding details
- **IDF bit_width**: computed from `ceil(log2(distinct_count))`, aligned to valid NoSplit widths {1,2,3,4,5,6,7,8,9,10,12,16,21,32}
- **IDFMETA u32_b**: must match IDF bit_width exactly — mismatch causes `QuerySystemError`
- **Dictionary order**: String columns use insertion order; numeric columns use sorted order
- **R$ INDEX**: +3 DATA_ID_OFFSET padding at positions 0-2; values are 1-based row indices into TO table
- **R$ RecordCount**: distinct FK values + 3 (not total FK rows)
- **H$ POS_TO_ID**: must use same sort order as encoder's dictionary
- **RLE encoding disabled**: pure bitpack used for IDF segments — slightly less space-efficient but correct

### Empty tables (`rows=[]`)

Supported as of 0.9.5 — a table with columns but no rows opens and queries in
PBI Desktop (`COUNTROWS` returns blank, `VALUES` returns an empty set). The
zero-row representation mirrors Desktop's own empty-table convention:

- Zero-row partition: `SegmentMapStorage` RecordCount=0, SegmentCount=1,
  RecordsPerSegment=0 (Type=3), Partition Type=4 / Mode=0 / DataView=3
- `AttributeHierarchyStorage.MaterializationType=2` with `DistinctDataCount=0`
  and no H$ system table (Desktop uses MatType=2 for its own zero-row table;
  MatType=3 is reserved for the RowNumber of a *populated* table)
- Empty **string** stores carry **no dictionary page** (`store_page_count=0`).
  A page with `allocation_size=0` has a null character buffer, which Analysis
  Services' string-store consistency check rejects
  (`PFE_XM_DBCC_STRINGSTORE_CORRUPT`) — a store with no strings has no page.

### Relationship semantics
The builder and every datamodel-edit path preserve and can author non-default
relationship traits (verified byte-for-byte against Power BI Desktop-authored
files, and round-tripped: files open with no repair prompt, correct glyphs in
Manage relationships):

| Trait | Encoding | Support |
|-------|----------|---------|
| Active / inactive | `IsActive` 1/0 (storage unchanged) | ✅ author + preserve |
| Cross-filter single / both | `CrossFilteringBehavior` 1/2 (storage unchanged) | ✅ author + preserve |
| Many-to-one (default) | `2→1`, single R$ index | ✅ |
| One-to-many | `1→2`, single R$ index | ✅ author + preserve |
| Many-to-many | `2→2`, **no** storage (`StorageID=0`, no R$ table) | ✅ author + preserve |
| One-to-one | `1→1`, cross-filter forced Both, **two** R$ indexes (`RelationshipStorageID` + `RelationshipStorage2ID`, a forward + reverse mirror R$ table) | ✅ author + preserve (verified byte-for-byte against Desktop; a 1:1 with only the single forward index fails to load) |

Before 0.9.10 any datamodel edit (add measure, modify column, …) silently reset
every relationship to active / single-direction / many-to-one; that data loss is
fixed — existing semantics now survive the rebuild.

### Editing models with calculated or measure-only tables
DataModel-edit tools come in two flavors:

- **Surgical** (edit the metadata in place, no rebuild) — work on **all** models:
  `pbix_datamodel_add_measure`, `modify_measure`, `remove_measure`,
  `modify_column`, `set_rls_role`, `modify_metadata`.
- **Rebuild** (regenerate the whole DataModel from scratch) — `add_relationship`,
  `remove_relationship`, `remove_table`, `add_field_parameter`,
  `add_calculation_group`, `set_table_data`, `update_table_rows`, `replace_value`.

**Measure-only container tables** (a table that holds only measures, e.g. a
`_Measures` table — no data columns) **are preserved** through a rebuild, so the
rebuild tools work on the very common "measures table + import tables" model
shape.

### Calculated columns across a rebuild

A rebuild regenerates VertiPaq from scratch, so a **calculated column**
(`Column.Type` 2 or 4) has to be RE-EVALUATED from its DAX. The evaluator is
deliberately narrow, and the rule is absolute: an expression it cannot
reproduce **exactly** is REFUSED, naming the reason, never materialised with a
guess. A wrong value stored in VertiPaq is invisible — the file opens fine and
every number is quietly wrong — which is worse than an error.

Reproduced exactly (each verified against the values Power BI Desktop itself
stored in the corpus files, which is exact ground truth):

- row-context expressions over the table's **own** columns —
  `fct[Sales] - fct[Cost]`, `IF(t[Qty] > 0, "Yes", "No")`, `ROUND(t[X], 2)`,
  `VAR ... RETURN ...`;
- plain whole-column aggregates over the table's own column (`SUM`, `MIN`,
  `MAX`, `AVERAGE`) — a calculated column has no filter context beyond its own
  row, so these really are the whole-column value on every row;
- the auto date/time accessor `X.[Date]` (the date part of X). The other
  variation parts (`.[Year]`, `.[Month]`, `.[Quarter]`, `.[Day]`, `.[MonthNo]`)
  map to locale-dependent display strings and are refused;
- **`LOOKUPVALUE`** — including a multi-column search, an alternate result, and
  a self-lookup. String matching is case-insensitive, matching Power BI's
  default collation. Rows that match with DIFFERENT values are an error in DAX,
  so the column is refused rather than picking one;
- **`RELATED`** — many-to-one only, across ACTIVE relationships, when exactly
  ONE path exists. Two paths (ambiguous without `USERELATIONSHIP`), no path,
  the wrong direction, an inactive relationship, or a bare `RELATED([Col])`
  that does not name its table are each refused;
- **`CALCULATE(<aggregate>, FILTER(<own table>, <predicate>))`**, where the
  predicate is a conjunction of comparisons against the table's own columns.
  `EARLIER` is supported. In a calculated column CALCULATE performs context
  transition and a FILTER over that same table replaces it, so this means
  "aggregate every row satisfying the predicate". The predicate is COMPILED
  into a hash index (equality terms) or a prefix aggregate over a sorted key
  (one inequality), so a 1.7M-row table costs one lookup per row rather than a
  table scan.

Still refused: the X-iterators (`SUMX`, `RANKX`, …), `PATH`/`PATHITEM`, `OR`
(`||`) inside a FILTER predicate, more than one inequality, `KEEPFILTERS`, an
aggregate over a DIFFERENT table, and `DISTINCTCOUNT`/`COUNT` outside the
compiled CALCULATE path.

**Calculated tables** are preserved rather than recomputed where their shape is
reproducible (`DATATABLE`, `GENERATESERIES`, `DISTINCT`, `VALUES`, `FILTER`,
`TOPN`, `ADDCOLUMNS`, a bare table reference); **field parameters** (a calc
table whose partition holds a `{("d", NAMEOF('T'[C]), n)}` tuple set) are
rebuilt and re-stamped. A rebuild that meets a shape it cannot reproduce
refuses with `MODEL_EDIT_UNSUPPORTED`, naming the offending column and pointing
at the surgical tools (which work on all models). Removing a special table
never blocks the edit — `pbix_datamodel_remove_table` always works.

**Calculation groups, translations and detail-rows** are carried across a
rebuild, along with named sets, query groups, aggregation wiring, refresh
policies and calendars. A calculation group is wired from both ends, so the
`[Table].CalculationGroupID` back-reference and the `Type=7` partition (which
must carry NO `QueryDefinition`) are restored too — verified by opening the
rebuilt file in Power BI Desktop and querying `INFO.CALCULATIONGROUPS()`.

**Known precision limit:** DateTime columns decode to a Python `datetime`
(microsecond resolution) while VertiPaq stores a double serial, so scaling a
timestamp to milliseconds (`[Timestamp] * 86400000`) can differ from Desktop in
the sub-microsecond digits.

### Other builder notes
- Fixed RowNumber GUID: 2662979B-1795-4F74-8F37-6A1BA8059B61
- Relationship direction: default many-to-one auto-detects Many/One sides
  (From=Many/fact, To=One/dimension); explicit cardinality or the preservation
  path keeps the caller's / source file's orientation verbatim
- Key annotations: PBI_IsFromSource (ObjectType=7), PBI_ResultType, SummarizationSetBy, PBI_QueryOrder, __PBI_TimeIntelligenceEnabled
- M expression navigation uses `Item` key (not `Name`) for MySQL/PostgreSQL

## Performance

The DAX engine operates on in-memory Python data structures. For large tables:
- Tables are loaded fully into memory as Python lists
- No columnar compression or VertiPaq-style optimization
- Expect degraded performance above ~100K rows
- Consider using filter context to limit evaluated data
