Metadata-Version: 2.4
Name: openqvd
Version: 1.0.0
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Requires-Dist: pyarrow>=12
Requires-Dist: openqvd[polars,pandas] ; extra == 'all'
Requires-Dist: pandas>=1.5 ; extra == 'pandas'
Requires-Dist: pyarrow>=12 ; extra == 'pandas'
Requires-Dist: polars>=0.19 ; extra == 'polars'
Provides-Extra: all
Provides-Extra: pandas
Provides-Extra: polars
Summary: Clean-room reader/writer for Qlik QVD files — Python bindings
Keywords: qvd,qlik,qlikview,qliksense,arrow,polars,pandas
Home-Page: https://sigilweaver.app/openqvd
Author-email: Nathan Riley <git@nathanriley.com>
License: AGPL-3.0-or-later
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/Sigilweaver/OpenQVD/blob/main/CHANGELOG.md
Project-URL: Homepage, https://sigilweaver.app/openqvd
Project-URL: Repository, https://github.com/Sigilweaver/OpenQVD

# OpenQVD

A free, open, clean-room specification and implementation of the Qlik QVD
binary file format, derived purely by binary analysis of publicly available
sample files. The goal is a Rust reader and writer that the data science
community can use without depending on any proprietary Qlik tooling.

The specification was developed against the
[QVD-Sources](https://github.com/Sigilweaver/QVD-Sources) corpus — a curated
collection of ~1,100 publicly available `.qvd` files gathered from GitHub.

## Status

Seven stages complete:

1. XML header and envelope structure. (Spec section 1.)
2. Per-field symbol table encoding. (Spec section 2.)
3. Bit-packed row index encoding. (Spec section 3.)
4. Validation against the full public corpus via a clean-room Python
   decoder.
5. Rust reader prototype (`crates/openqvd`) with edge-case tests.
6. Writer + semantic round-trip tests.
7. Python bindings (`crates/openqvd-py`) — PyArrow, Polars, Pandas.

See `SPEC.md` for the current specification and `NOTES.md` for the
working log of observations.

### Rust usage

```toml
# Cargo.toml
[dependencies]
openqvd = "1"

# Enable Arrow integration (PyArrow, RecordBatch, type inference):
openqvd = { version = "1", features = ["arrow"] }
```

```rust
use openqvd::Qvd;

let qvd = Qvd::from_path("data.qvd").unwrap();
println!("{} rows", qvd.num_rows());
for row in qvd.rows() {
    // row: Vec<Option<Value>>
}
```

### Reader

The Rust reader parses **1,044 of 1,047** valid public QVD samples. The
three remaining files are deliberately-corrupted test fixtures from
third-party projects (two named `damaged.qvd`, one with invalid UTF-8).
10 unit + integration tests cover bias-based NULL, 2+6 bit packing,
zero-width fields, every symbol type byte, unknown-type rejection,
overlapping bit-fields rejection, inconsistent root `Length`
rejection, and the LF-terminator header variant.

### Writer

A compliant writer is implemented in `crates/openqvd::writer`. Running
`read -> write -> read` over the entire corpus yields **1,093 of 1,093
valid files semantically equivalent** (same row count, same field
names, byte-for-byte equal cell values). 9 writer tests cover NULL
handling, all five symbol types, zero-width collapse for constant
columns, 500-distinct wide columns, NUL-in-string rejection,
uneven-column rejection, and deterministic output.

### Python bindings

`crates/openqvd-py` is a [maturin](https://maturin.rs/) mixed-layout
package that exposes a pure-Python API on top of the Rust library.

**Install (development)**

```sh
cd crates/openqvd-py
uv venv .venv && source .venv/bin/activate
uv pip install maturin pyarrow polars pandas
maturin develop
```

**Usage**

```python
import openqvd

# Read as a PyArrow Table
table = openqvd.read("data.qvd")
table = openqvd.read("data.qvd", columns=["OrderId", "Amount"])

# Predicate pushdown (filtering at the Rust level, before Arrow conversion)
table = openqvd.read("data.qvd", filters=[
    {"column": "Region", "op": "eq", "value": "West"},
    {"column": "Status", "op": "is_in", "value": ["Open", "Pending"]},
    {"column": "Notes",  "op": "is_not_null"},
])

# Inspect metadata only (no row decoding)
info = openqvd.schema("data.qvd")
print(info.table_name, info.num_rows)
print([f.name for f in info.fields])

# Write from a PyArrow Table
openqvd.write(table, "out.qvd")
openqvd.write(table, "out.qvd", table_name="Orders")

# Polars (import registers pl.read_qvd, pl.scan_qvd, df.qvd.write)
import openqvd.polars
import polars as pl

df = pl.read_qvd("data.qvd")
lf = pl.scan_qvd("data.qvd", columns=["A", "B"])
df = pl.read_qvd("data.qvd", filters=[{"column": "A", "op": "eq", "value": "x"}])
df.qvd.write("out.qvd")

# Pandas (via PyArrow)
df = openqvd.read("data.qvd").to_pandas()
```

The Python bindings read **1,044 of 1,047** valid corpus files (99.7%),
matching the Rust reader baseline. The 3 failures are deliberately-
corrupted test fixtures.

**Arrow type mapping**

| QVD NumberFormat/Type | Arrow type |
|---|---|
| `DATE` | `Date32` (Qlik epoch → Unix epoch) |
| `TIMESTAMP` | `Timestamp(Microsecond, None)` |
| `TIME` | `Duration(Microsecond)` |
| Int / DualInt symbols | `Int64` |
| Float / DualFloat symbols | `Float64` |
| String symbols | `LargeUtf8` |
| Empty symbol table | `Null` |

### CLI

The `openqvd` binary provides end-user tooling:

```
openqvd stat <file>           # header summary (fields, widths, rows)
openqvd head <file> [--rows N]  # first N rows
openqvd csv  <file>           # every row as tab-separated text
openqvd json <file>           # one JSON object per row
openqvd rewrite <in> <out>    # read then re-serialise through the writer
```

## Non-goals

- Executing, shipping, or linking any proprietary Qlik code.
- Reading closed or encrypted QVD variants (if they exist).
- Parsing QVW, QVF, or QVS files (those are separate formats).

## License

AGPL-3.0-or-later. See `LICENSE`.

