Metadata-Version: 2.4
Name: pyivt
Version: 0.0.1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Dist: pyarrow>=14
Requires-Dist: pytest>=7 ; extra == 'dev'
Requires-Dist: ruff>=0.6 ; extra == 'dev'
Requires-Dist: maturin>=1.5,<2 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Read Statistics Canada Beyond 20/20 .ivt tables: metadata and Apache Arrow export
Keywords: ivt,beyond-2020,statistics-canada,statcan,arrow,pyarrow
Author: Nate Babyak
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/natebabyak/pyivt/blob/main/CHANGELOG.md
Project-URL: Homepage, https://github.com/natebabyak/pyivt
Project-URL: Issues, https://github.com/natebabyak/pyivt/issues
Project-URL: Repository, https://github.com/natebabyak/pyivt

# pyivt

`pyivt` is a library for Statistics Canada Beyond 20/20 `.ivt` files, written in Rust. It is designed to be minimal and easily adaptable for existing codebases. Key features are:

- **Fast**: Orders of magnitude faster than GUI automation for metadata (see local `benchmarks/` if you want timings)
- **Python**: Easily integrates into modern data workflows with a minimal API
- **Typed**: Produces fully type-safe and predictable metadata
- **Interoperable**: Exports `.ivt` file data to Apache Arrow

## Installation

```bash
uv add pyivt
# or: uv pip install pyivt / pip install pyivt
```

Requires Python 3.10+ and a platform wheel (abi3) or a local Rust toolchain for source builds.

### Development

```bash
uv sync --extra dev
uv run python tests/scripts/download.py   # StatCan fixtures for pytest
uv run pytest
# or: uv run maturin develop --release
```

## Usage

### Quick Start

Open an IVT file, read its metadata, and write its data to a Parquet file:

```python
import pyarrow.parquet as pq
import pyivt

ivt = pyivt.open("data.ivt")
print(ivt.metadata.title.en)
pq.write_table(ivt.to_arrow(), "data.parquet")
```

### Opening IVT Files

`pyivt.open` accepts a path, bytes-like buffer, or binary file object. Summary metadata and layout are parsed immediately; cell values decode only when you call `to_arrow()`.

#### From Path

```python
import pyivt

ivt = pyivt.open("data.ivt")
```

#### From URL

```python
import requests

import pyivt

url = "https://example.com/data.ivt"
res = requests.get(url)
res.raise_for_status()

ivt = pyivt.open(res.content)
```

### Reading Metadata

`ivt.metadata` is a typed `IvtMetadata` object. Summary fields are eager; heavy fields load on first access.

| Beyond 20/20 Summary | `pyivt` Metadata             |
| -------------------- | ---------------------------- |
| Title                | `title.en`, `title.fr`       |
| Author               | `author.en`, `author.fr`     |
| Category             | `category.en`, `category.fr` |
| Keywords             | `keywords.en`, `keywords.fr` |
| Notes                | `notes.en`, `notes.fr`       |
| Number of Datapoints | `number_of_datapoints`       |
| Time Series          | `time_series` (`.frequency`, `.start`, `.end`) or `None` |
| Read-Only            | `read_only`                  |
| Dimensions           | `dimensions`                 |
| Geographies          | `geographies`                |
| Footnotes            | `footnotes`                  |
| Missing Values       | `missing_values`             |

`geographies` is a `Geographies` object (`member_id`, `geo_label`, `geo_uid`, …), not a dict.

```python
import pyivt

ivt = pyivt.open("data.ivt")
print(ivt.metadata.product_id, ivt.metadata.title.en)
```

### Exporting Data

`Ivt.to_arrow()` returns an Apache Arrow table. Keyword arguments:

| Argument | Default | Meaning |
| -------- | ------- | ------- |
| `complete` | `True` | Expand the published grid; set `False` for sparse decoded cells only |
| `max_cells` | `100_000_000` | Raise if the complete grid would exceed this many cells |
| `labels` | `True` | Include dimension label columns |
| `language` | `"en"` | Label language (`"en"` or `"fr"`) |

```python
import polars as pl
import pyarrow.parquet as pq

table = ivt.to_arrow()

# pandas
df = table.to_pandas()

# Parquet
pq.write_table(table, "data.parquet")

# Polars
df = pl.from_arrow(table)
```

## License

`pyivt` is licensed under the [MIT License](LICENSE).

## Acknowledgements

- [canivt](https://github.com/mountainMath/canivt)
- [REA: Reverse Engineer Anything](https://github.com/morluto/rea)

