Metadata-Version: 2.4
Name: qvd-parser
Version: 0.2.1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: pydantic>=2.0 ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: pytest-benchmark>=4.0 ; extra == 'dev'
Requires-Dist: pydantic>=2.0 ; extra == 'pydantic'
Provides-Extra: dev
Provides-Extra: pydantic
License-File: LICENSE
Summary: Fast QVD file parser — Python bindings for qvd-parser (Rust)
Keywords: qvd,qlik,qlikview,qliksense,parser
Home-Page: https://github.com/nicolasstefaniuk/qvd-parser
Author-email: Nicolas Stefaniuk <287317062+nicolasstefaniuk@users.noreply.github.com>
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/nicolasstefaniuk/qvd-parser
Project-URL: Issues, https://github.com/nicolasstefaniuk/qvd-parser/issues
Project-URL: Repository, https://github.com/nicolasstefaniuk/qvd-parser

# qvd-parser

[![Crates.io](https://img.shields.io/crates/v/qvd-parser.svg)](https://crates.io/crates/qvd-parser)
[![PyPI](https://img.shields.io/pypi/v/qvd-parser.svg)](https://pypi.org/project/qvd-parser/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Fast, correct QVD (QlikView Data) file parser written in Rust.

Reads the binary format used by QlikView and Qlik Sense to store compressed
data tables. Exposes a Rust API, a Python API (via PyO3), and a CLI. Export
targets: JSON, CSV, Parquet.

> **Note:** QVD is a proprietary file format associated with QlikView and
> Qlik Sense. This project is not affiliated with or endorsed by Qlik.
> See [Legal & Interoperability](#legal--interoperability) for details.

## Workspace structure

```
qvd-parser/
├── core/               # Core parsing library — QvdFile, QvdValue, RowIter
├── convert/            # Export backends: Parquet, JSON, CSV
├── cli/                # Command-line interface (qvd-parser-cli binary)
└── bindings-python/    # Python bindings via PyO3 + maturin
```

## Prerequisites

- Rust toolchain (1.75+): https://rustup.rs
- For Python bindings: `pip install maturin`

## Build

```bash
# Build all crates
cargo build --workspace

# Build release (recommended for performance measurement)
cargo build --release --workspace

# Run all Rust unit tests
cargo test --workspace

# Run integration tests (requires QVD fixtures — see below)
cargo test --workspace --features integration-tests

# Build the Python extension in-place (for development)
maturin develop

# Run Python binding tests
pytest bindings-python/tests/
```

## CLI usage

```bash
# Schema inspection
cargo run -p qvd-parser-cli -- schema --input data.qvd

# Export to JSON
cargo run -p qvd-parser-cli -- export --input data.qvd --format json --output out.json

# Export to CSV (semicolon delimiter)
cargo run -p qvd-parser-cli -- export --input data.qvd --format csv --output out.csv --csv-delimiter ";"

# Export to Parquet (verbose timing breakdown)
cargo run --release -p qvd-parser-cli -- export --input data.qvd --format parquet --output out.parquet --verbose
```

### Parquet export options

| Flag | Default | Description |
|---|---|---|
| `--cast FIELD=TYPE` | — | Force a column to a specific Arrow type. Repeat for multiple fields. Valid types: `INT32`, `INT64`, `FLOAT32`, `FLOAT64`, `STRING`, `DATE32`, `TIMESTAMP_MS`, `BOOLEAN`. Takes priority over `--schema-overrides` on conflict. |
| `--schema-overrides FILE` | — | Path to a JSON file containing per-field type overrides. Same `{"FieldName": "TYPE"}` format as `--cast` but file-based. Inline `--cast` values take priority on conflict. |
| `--fallback-type TYPE` | `STRING` | Type used when inference is inconclusive — i.e. a column's symbol table is all-null/empty and `NumberFormat` carries no recognised hint either. Same valid types as `--cast`. Never overrides a genuine text result from the symbol scan. |
| `--row-group-size N` | 100000 | Rows per Parquet row group |
| `--strict-first` | off | Fail immediately on the first cast error instead of writing null. Mutually exclusive with `--strict-all`. |
| `--strict-all` | off | Run the full export collecting every cast error, then fail with the complete list if any occurred. The output file is still fully written (failed values as null) even though the command exits with an error. Mutually exclusive with `--strict-first`. |
| `--encoding auto\|plain\|dictionary` | `auto` | Arrow array encoding strategy. **CLI-only, experimental** — not exposed in Python bindings (see performance notes below). |
| `--no-statistics` | off | Disable column min/max statistics (~20% faster export, less efficient predicate pushdown for consumers). **CLI-only, experimental** — not exposed in Python bindings. |

By default (no `--strict-first`/`--strict-all`), cast failures are written
as `null` and reported as warnings on stderr after the export completes —
nothing is silently lost without a trace:

```
warning: 3 value(s) failed to cast and were written as null:
  field 'Amount' row 42: cannot parse 'N/A' as FLOAT64: invalid float literal
  field 'Amount' row 109: cannot parse 'N/A' as FLOAT64: invalid float literal
  field 'Date' row 7: cannot convert text 'unknown' to DATE32 — use a numeric day-offset value
```

```bash
# Force specific column types
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --cast DateField=DATE32 --cast Amount=FLOAT64

# Use a JSON override file
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --schema-overrides overrides.json

# Both combined — inline --cast takes priority on conflict
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --schema-overrides overrides.json --cast DateField=TIMESTAMP_MS

# Fail with a complete error report if any value fails to cast
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --strict-all

# All-null/empty columns default to STRING; force INT64 instead
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --fallback-type INT64
```

### Verbose timing output

`--verbose` shows a per-phase breakdown of both parsing and export:

```
[qvd] export: data.qvd
  load header+symbols+rows        124 ms  [140 fields  119820 rows  126 B/rec]
    header                          1 ms
    symbols                        23 ms
    rows                           98 ms
  export parquet                  555 ms  [out.parquet]
    type inference                  0 ms
    index transpose                43 ms
    cast to arrow                 166 ms
    parquet encode                296 ms
    parquet close                  49 ms
```

### Type inference

For each column without an explicit `--cast`/`schema_overrides` entry, the
target Arrow type is resolved in this order:

1. **Symbol scan** — the full symbol table is scanned; the dominant
   non-null variant wins (`Int64`/`Float64` only if zero text symbols are
   present — any text presence forces `STRING`, since a numeric type would
   silently null out non-numeric values).
2. **`NumberFormat.type_name` hint** — used only when the symbol scan is
   inconclusive (column is all-null or empty). `NumberFormat` is a Qlik
   *display format* declaration, not a data-type guarantee — a pure-text
   field can legally carry `<Type>REAL</Type>` — so it is never trusted
   over actual stored data.
3. **`--fallback-type`** — last resort, used only when both the symbol scan
   and the `NumberFormat` hint are inconclusive. Defaults to `STRING`.

Cast failures (e.g. a `--cast Field=INT64` override on a column containing
non-numeric text) do not abort the export by default — see `--strict-first`
/ `--strict-all` above to change that, and the warning output shown there
for how failures are reported.

## Python usage

```python
import qvd_parser

# From a file path (Rust opens the file directly)
f = qvd_parser.QvdFile.open("data.qvd")
print(f.table_name, f.row_count, f.field_names())

for row in f.rows():
    print(row)  # list of Python-native values (int, float, str, tuple, None)

# Export
f.to_parquet("out.parquet")
f.to_parquet("out.parquet", cast_overrides={"DateField": "DATE32", "Amount": "FLOAT64"})
f.to_parquet("out.parquet", schema_overrides_file="overrides.json")
f.to_parquet("out.parquet", strict_mode="strict_first")  # raise on first cast error
f.to_parquet("out.parquet", strict_mode="strict_all")    # raise with the full error list
f.to_parquet("out.parquet", fallback_type="INT64")       # all-null columns default to INT64 instead of STRING
f.to_json("out.json")
f.to_csv("out.csv", delimiter=";")

# to_parquet() returns a CastWarnings object (default strict_mode="silent"):
# empty/falsy if every value cast successfully, otherwise inspect .entries
warnings = f.to_parquet("out.parquet", cast_overrides={"Amount": "FLOAT64"})
if warnings:
    for w in warnings.entries:
        print(w)  # "field 'Amount' row 42: cannot parse 'N/A' as FLOAT64: ..."
```

### Parsing from in-memory bytes

When the QVD is already in memory — downloaded from a signed URL, read from
object storage, or loaded via any other source that does not produce a file
path — use `from_bytes()`:

```python
# From a signed URL (no temporary file needed)
import requests
data = requests.get(signed_url).content
f = qvd_parser.QvdFile.from_bytes(data)

# AWS S3
data = s3_client.get_object(Bucket="my-bucket", Key="data.qvd")["Body"].read()
f = qvd_parser.QvdFile.from_bytes(data)

# Python file handle (equivalent to open(), Python controls the lifecycle)
with open("data.qvd", "rb") as fh:
    f = qvd_parser.QvdFile.from_bytes(fh.read())
```

All methods available on `open()` results are available on `from_bytes()`
results: `rows()`, `to_dict()`, `to_parquet()`, `to_json()`, `to_csv()`, etc.

> **Note on memory usage:** `from_bytes()` requires the entire QVD to be in
> RAM. This is unavoidable when the source is an in-memory buffer. If you
> have a file path and memory is a concern, prefer `open()` — it lets the OS
> manage buffering.

### Python version compatibility

Tested against Python 3.13. CPython 3.11 and 3.12 are declared compatible
for Snowflake runtime support but are not covered by CI.

## Test fixtures

Integration tests require real QVD files generated by Qlik — synthetic QVD
files are not possible due to the proprietary bit-packing format.

Place fixture files in `fixtures/` at the workspace root:

| File | Content |
|---|---|
| `test-data-minimal.qvd` | 2 columns (`id: Int`, `label: Text`), 2 rows: `[1, "test1"], [2, "test2"]` — primary fixture for Rust and Python tests |
| `all-nulls.qvd` | 2 columns, `nullable_col` entirely NULL — 3 rows |
| `all-types.qvd` | 6 columns covering all `QvdValue` variants: Int, Double, Text, DualInt, DualDouble, Null |
| `single-row.qvd` | Exactly 1 row — iterator and size_hint edge case |
| `empty-table.qvd` | 0 rows, schema present — 2 columns |
| `large-integers.qvd` | Values at `i32::MIN`/`i32::MAX` and byte/word boundaries |
| `unicode-strings.qvd` | Multi-byte UTF-8 strings: accented Latin characters, CJK |
| `high-cardinality.qvd` | 100 rows, `unique_label` with 100 distinct values |
| `wide-record.qvd` | `RecordByteSize > 16` — regression fixture for the u128 overflow bug |
| `mixed-types.qvd` | Heterogeneous symbol table: Int, Double, Text, NULL in the same column |
| `bench-symbols.qvd` | 1 000 rows × 3 fields, high-cardinality symbol tables — benchmark fixture |
| `bench-rows.qvd` | 5 000 rows × 8 fields, mixed cardinality, all QvdValue variants — benchmark fixture |

All fixtures must be generated with Qlik (QlikView or Qlik Sense) using
`generate-fixtures.qvs`. Synthetic QVD generation is not possible due to the
proprietary bit-packing format.

Tests that rely on missing fixtures are skipped at runtime when
`--features integration-tests` is active.

## Feature flags

| Flag | Crate | Effect |
|---|---|---|
| `integration-tests` | `core`, `convert` | Enable tests that require fixture files |
| `serde` | `core` | Derive `Serialize`/`Deserialize` on public types |

### Python binding entry points

| Method | Input | Use case |
|---|---|---|
| `QvdFile.open(path: str)` | File path | Rust opens and buffers the file — preferred when a path is available |
| `QvdFile.from_bytes(data: bytes \| bytearray \| memoryview)` | In-memory bytes | QVD already in RAM (signed URL, object storage, `BytesIO`) |

---

## Release workflow

This project uses two separate repositories:

- **GitLab** (private) — daily development, full commit history
- **GitHub** (public) — published releases only, single commit per version

GitHub receives only validated, committed code from GitLab. The full
development history never appears on GitHub.

### Publishing a new release (Windows)

Ensure the version to publish is committed and pushed on GitLab `main`:

```powershell
# 1. From the GitLab workspace, export the clean content
cd C:\path\to\qvd-parser
git checkout main
git pull origin main
git archive HEAD | tar -x -C C:\path\to\qvd-parser-public

# 2. Commit and push to GitHub
cd C:\path\to\qvd-parser-public
git add .
git commit -m "Release vX.Y.Z"
git push github main
```

`git archive HEAD` exports exactly the content of the last commit, respecting
`.gitignore`. Files not committed (build artifacts, fixtures, `.venv`) are
automatically excluded.

### GitHub repository setup (once)

```powershell
mkdir C:\path\to\qvd-parser-public
cd C:\path\to\qvd-parser-public
git init
git remote add github git@github.com:nicolasstefaniuk/qvd-parser.git
git config user.email "287317062+nicolasstefaniuk@users.noreply.github.com"
git config user.name "Nicolas Stefaniuk"
```

---

## Performance — Parquet export investigation

### Context

QVD files use a columnar symbol table + row-major bit-packed index buffer.
Parsing is inherently row-major (impossible to read one column without traversing
all rows). Parquet is column-major. This impedance mismatch is the core
performance challenge.

### Measurements (release build, 140 fields × 119 820 rows × 16.5 MB QVD)

```
load header+symbols+rows    ~120 ms
  header                       1 ms
  symbols                     23 ms
  rows (bit-unpack)           95 ms

export parquet              ~650 ms
  type inference               0 ms
  index transpose             43 ms   ← column_indices_range
  cast to arrow              160 ms   ← QvdValue → Arrow arrays
  parquet encode             400 ms   ← RLE/dictionary encoding (Arrow writer)
  parquet close               48 ms
```

Ratio parsing:export ≈ 1:5. The bottleneck is the Arrow/Parquet writer, not our code.

### Hypotheses tested

**1. Cache thrashing on strided index access (fixed)**

The original export collected per-column indices by striding through the
row-major buffer (`n_fields × 4` bytes per step), causing cache misses on
wide files. Fixed by adding `QvdFile::column_indices_range()` in `core`:
a single sequential pass that scatters into `n_fields` contiguous vectors.
Measured gain: ~15% on `index transpose`.

**2. Dictionary encoding for string columns (implemented, marginal gain)**

With only 11–15 distinct values per column out of 119 820 rows, passing
pre-built `DictionaryArray<Int32, Utf8>` to the Parquet writer should
avoid per-row string copies and skip the encoder's internal deduplication.

Result: ~6% total gain in release, negligible. The Arrow writer (arrow-rs 53.x)
still does substantial work on `DictionaryArray`. Additionally, numeric
dictionary arrays (`DictionaryArray<Int32, Int64>` etc.) are not supported
by `ArrowWriter` in this version and panic at runtime — restricted to `Utf8` only.

Available as `--encoding dictionary` for experimentation. Not recommended for
production use until arrow-rs improves dictionary support.

**3. Column statistics (implemented, ~20% gain)**

Parquet column statistics (min/max per page) cost ~130 ms on this file.
They are enabled by default because they allow query engines to skip row
groups during reads, which typically saves far more time than the export cost.

Disabling them with `--no-statistics` saves ~20% export time. Use when
export speed is critical and the consumer does not rely on statistics.

### Conclusion

The 5× ratio between parsing and Parquet export is inherent to the Arrow
writer, not to our code. The practical levers available in our layer are:

| Optimisation | Gain | Status |
|---|---|---|
| `column_indices_range` (sequential scatter) | ~15% index transpose | Enabled by default |
| `--no-statistics` | ~20% parquet encode | Opt-in flag |
| `--encoding dictionary` | <10% on Utf8 columns | Experimental, opt-in |

A more significant improvement would require either a lower-level Parquet
crate (e.g. `parquet2`) or parallelising column encoding — both are
substantial changes outside the current scope.

---

## Legal & Interoperability

### Trademark notice

"Qlik", "QlikView", and "Qlik Sense" are registered trademarks of Qlik
Technologies Inc. The file extension "QVD" is a technical descriptor for
the file format, not a registered trademark.

This project is not affiliated with, endorsed by, or sponsored by Qlik in
any way. The name `qvd-parser` is used solely to describe what the library
does — reading files in the QVD format — and constitutes nominative
descriptive use, not trademark infringement.

### Reverse engineering and interoperability

This library was developed by observing the structure of QVD files produced
by Qlik software. No Qlik source code was copied, decompiled, or
incorporated. The parser is an independent implementation.

Reading a file format for the purpose of interoperability is explicitly
permitted under applicable law:

- **European Union** — Directive 2009/24/EC on the legal protection of
  computer programs, Article 6, permits decompilation and reverse engineering
  for interoperability purposes, notwithstanding any contractual restriction
  to the contrary.
- **United States** — 17 U.S.C. § 1201(f) (DMCA) provides a specific
  exemption for reverse engineering undertaken to achieve interoperability.
  This is supported by established case law (*Sega Enterprises Ltd. v.
  Accolade, Inc.*, 977 F.2d 1510; *Sony Computer Entertainment, Inc. v.
  Connectix Corp.*, 203 F.3d 596).

File formats — as distinct from software implementations — are not protected
by copyright as a matter of EU and US law (*SAS Institute Inc. v. World
Programming Ltd*, CJEU C-406/10, 2012).

Enterprise users who require legal sign-off before adopting a dependency are
encouraged to share this section with their legal team.

### License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for
the full text.

Copyright (c) 2025 Nicolas Stefaniuk
