Metadata-Version: 2.4
Name: qvd-parser
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: pydantic>=2.0 ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: pydantic>=2.0 ; extra == 'pydantic'
Provides-Extra: dev
Provides-Extra: pydantic
License-File: LICENSE
Summary: Fast QVD file parser — Python bindings for qvd-parser (Rust)
Keywords: qvd,qlik,qlikview,qliksense,parser
Home-Page: https://github.com/nicolasstefaniuk/qvd-parser
Author-email: Nicolas Stefaniuk <287317062+nicolasstefaniuk@users.noreply.github.com>
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/nicolasstefaniuk/qvd-parser
Project-URL: Issues, https://github.com/nicolasstefaniuk/qvd-parser/issues
Project-URL: Repository, https://github.com/nicolasstefaniuk/qvd-parser

# qvd-parser

[![Crates.io](https://img.shields.io/crates/v/qvd-parser.svg)](https://crates.io/crates/qvd-parser)
[![PyPI](https://img.shields.io/pypi/v/qvd-parser.svg)](https://pypi.org/project/qvd-parser/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Fast, correct QVD (QlikView Data) file parser written in Rust.

Reads the binary format used by QlikView and Qlik Sense to store compressed
data tables. Exposes a Rust API, a Python API (via PyO3), and a CLI. Export
targets: JSON, CSV, Parquet.

> **Note:** QVD is a proprietary file format associated with QlikView and
> Qlik Sense. This project is not affiliated with or endorsed by Qlik.
> See [Legal & Interoperability](#legal--interoperability) for details.

## Workspace structure

```
qvd-parser/
├── core/               # Core parsing library — QvdFile, QvdValue, RowIter
├── convert/            # Export backends: Parquet, JSON, CSV
├── cli/                # Command-line interface (qvd-parser-cli binary)
└── bindings-python/    # Python bindings via PyO3 + maturin
```

## Prerequisites

- Rust toolchain (1.75+): https://rustup.rs
- For Python bindings: `pip install maturin`

## Build

```bash
# Build all crates
cargo build --workspace

# Build release (recommended for performance measurement)
cargo build --release --workspace

# Run all Rust unit tests
cargo test --workspace

# Run integration tests (requires QVD fixtures — see below)
cargo test --workspace --features integration-tests

# Build the Python extension in-place (for development)
maturin develop

# Run Python binding tests
pytest bindings-python/tests/
```

## CLI usage

```bash
# Schema inspection
cargo run -p qvd-parser-cli -- schema --input data.qvd

# Export to JSON
cargo run -p qvd-parser-cli -- export --input data.qvd --format json --output out.json

# Export to CSV (semicolon delimiter)
cargo run -p qvd-parser-cli -- export --input data.qvd --format csv --output out.csv --csv-delimiter ";"

# Export to Parquet (verbose timing breakdown)
cargo run --release -p qvd-parser-cli -- export --input data.qvd --format parquet --output out.parquet --verbose
```

### Parquet export options

| Flag | Default | Description |
|---|---|---|
| `--row-group-size N` | 100000 | Rows per Parquet row group |
| `--encoding auto\|plain\|dictionary` | `auto` | Arrow array encoding strategy (see below) |
| `--no-statistics` | off | Disable column min/max statistics (~20% faster export, less efficient predicate pushdown for consumers) |
| `--strict` | off | Fail immediately on cast errors instead of writing null |

### Verbose timing output

`--verbose` shows a per-phase breakdown of both parsing and export:

```
[qvd] export: data.qvd
  load header+symbols+rows        124 ms  [140 fields  119820 rows  126 B/rec]
    header                          1 ms
    symbols                        23 ms
    rows                           98 ms
  export parquet                  555 ms  [out.parquet]
    type inference                  0 ms
    index transpose                43 ms
    cast to arrow                 166 ms
    parquet encode                296 ms
    parquet close                  49 ms
```

## Python usage

```python
import qvd_parser

f = qvd_parser.QvdFile.open("data.qvd")
print(f.table_name, f.row_count, f.field_names())

for row in f.rows():
    print(row)  # list of Python-native values (int, float, str, tuple, None)

# Export
f.to_parquet("out.parquet")
f.to_json("out.json")
f.to_csv("out.csv", delimiter=";")
```

### Python version compatibility

Tested against Python 3.13. CPython 3.11 and 3.12 are declared compatible
for Snowflake runtime support but are not covered by CI.

## Test fixtures

Integration tests require real QVD files generated by Qlik — synthetic QVD
files are not possible due to the proprietary bit-packing format.

Place fixture files in `fixtures/` at the workspace root:

| File | Content |
|---|---|
| `test_data01.qvd` | 2 columns (`id: Int`, `label: Text`), 2 rows: `[1, "test1"], [2, "test2"]` |

Tests that rely on missing fixtures are either skipped or marked `#[ignore]`.

## Feature flags

| Flag | Crate | Effect |
|---|---|---|
| `integration-tests` | `core`, `convert` | Enable tests that require fixture files |
| `serde` | `core` | Derive `Serialize`/`Deserialize` on public types |

---

## Release workflow

This project uses two separate repositories:

- **GitLab** (private) — daily development, full commit history
- **GitHub** (public) — published releases only, single commit per version

GitHub receives only validated, committed code from GitLab. The full
development history never appears on GitHub.

### Publishing a new release (Windows)

Ensure the version to publish is committed and pushed on GitLab `main`:

```powershell
# 1. From the GitLab workspace, export the clean content
cd C:\path\to\qvd-parser
git checkout main
git pull origin main
git archive HEAD | tar -x -C C:\path\to\qvd-parser-public

# 2. Commit and push to GitHub
cd C:\path\to\qvd-parser-public
git add .
git commit -m "Release vX.Y.Z"
git push github main
```

`git archive HEAD` exports exactly the content of the last commit, respecting
`.gitignore`. Files not committed (build artifacts, fixtures, `.venv`) are
automatically excluded.

### GitHub repository setup (once)

```powershell
mkdir C:\path\to\qvd-parser-public
cd C:\path\to\qvd-parser-public
git init
git remote add github git@github.com:nicolasstefaniuk/qvd-parser.git
git config user.email "287317062+nicolasstefaniuk@users.noreply.github.com"
git config user.name "Nicolas Stefaniuk"
```

---

## Performance — Parquet export investigation

### Context

QVD files use a columnar symbol table + row-major bit-packed index buffer.
Parsing is inherently row-major (impossible to read one column without traversing
all rows). Parquet is column-major. This impedance mismatch is the core
performance challenge.

### Measurements (release build, 140 fields × 119 820 rows × 16.5 MB QVD)

```
load header+symbols+rows    ~120 ms
  header                       1 ms
  symbols                     23 ms
  rows (bit-unpack)           95 ms

export parquet              ~650 ms
  type inference               0 ms
  index transpose             43 ms   ← column_indices_range
  cast to arrow              160 ms   ← QvdValue → Arrow arrays
  parquet encode             400 ms   ← RLE/dictionary encoding (Arrow writer)
  parquet close               48 ms
```

Ratio parsing:export ≈ 1:5. The bottleneck is the Arrow/Parquet writer, not our code.

### Hypotheses tested

**1. Cache thrashing on strided index access (fixed)**

The original export collected per-column indices by striding through the
row-major buffer (`n_fields × 4` bytes per step), causing cache misses on
wide files. Fixed by adding `QvdFile::column_indices_range()` in `core`:
a single sequential pass that scatters into `n_fields` contiguous vectors.
Measured gain: ~15% on `index transpose`.

**2. Dictionary encoding for string columns (implemented, marginal gain)**

With only 11–15 distinct values per column out of 119 820 rows, passing
pre-built `DictionaryArray<Int32, Utf8>` to the Parquet writer should
avoid per-row string copies and skip the encoder's internal deduplication.

Result: ~6% total gain in release, negligible. The Arrow writer (arrow-rs 53.x)
still does substantial work on `DictionaryArray`. Additionally, numeric
dictionary arrays (`DictionaryArray<Int32, Int64>` etc.) are not supported
by `ArrowWriter` in this version and panic at runtime — restricted to `Utf8` only.

Available as `--encoding dictionary` for experimentation. Not recommended for
production use until arrow-rs improves dictionary support.

**3. Column statistics (implemented, ~20% gain)**

Parquet column statistics (min/max per page) cost ~130 ms on this file.
They are enabled by default because they allow query engines to skip row
groups during reads, which typically saves far more time than the export cost.

Disabling them with `--no-statistics` saves ~20% export time. Use when
export speed is critical and the consumer does not rely on statistics.

### Conclusion

The 5× ratio between parsing and Parquet export is inherent to the Arrow
writer, not to our code. The practical levers available in our layer are:

| Optimisation | Gain | Status |
|---|---|---|
| `column_indices_range` (sequential scatter) | ~15% index transpose | Enabled by default |
| `--no-statistics` | ~20% parquet encode | Opt-in flag |
| `--encoding dictionary` | <10% on Utf8 columns | Experimental, opt-in |

A more significant improvement would require either a lower-level Parquet
crate (e.g. `parquet2`) or parallelising column encoding — both are
substantial changes outside the current scope.

---

## Legal & Interoperability

### Trademark notice

"Qlik", "QlikView", and "Qlik Sense" are registered trademarks of Qlik
Technologies Inc. The file extension "QVD" is a technical descriptor for
the file format, not a registered trademark.

This project is not affiliated with, endorsed by, or sponsored by Qlik in
any way. The name `qvd-parser` is used solely to describe what the library
does — reading files in the QVD format — and constitutes nominative
descriptive use, not trademark infringement.

### Reverse engineering and interoperability

This library was developed by observing the structure of QVD files produced
by Qlik software. No Qlik source code was copied, decompiled, or
incorporated. The parser is an independent implementation.

Reading a file format for the purpose of interoperability is explicitly
permitted under applicable law:

- **European Union** — Directive 2009/24/EC on the legal protection of
  computer programs, Article 6, permits decompilation and reverse engineering
  for interoperability purposes, notwithstanding any contractual restriction
  to the contrary.
- **United States** — 17 U.S.C. § 1201(f) (DMCA) provides a specific
  exemption for reverse engineering undertaken to achieve interoperability.
  This is supported by established case law (*Sega Enterprises Ltd. v.
  Accolade, Inc.*, 977 F.2d 1510; *Sony Computer Entertainment, Inc. v.
  Connectix Corp.*, 203 F.3d 596).

File formats — as distinct from software implementations — are not protected
by copyright as a matter of EU and US law (*SAS Institute Inc. v. World
Programming Ltd*, CJEU C-406/10, 2012).

Enterprise users who require legal sign-off before adopting a dependency are
encouraged to share this section with their legal team.

### License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for
the full text.

Copyright (c) 2025 Nicolas Stefaniuk

