Metadata-Version: 2.4
Name: polars-gdx
Version: 0.3.1
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering
Requires-Dist: polars>=1.0
Requires-Dist: pyarrow>=11
Requires-Dist: pytest>=8 ; extra == 'test'
Provides-Extra: test
License-File: LICENSE
License-File: crates/gdx-sys/third_party/gdx/LICENSE
License-File: crates/gdx-sys/third_party/gdx/zlib/LICENSE
Summary: Lazy, prefiltered Polars reader for GAMS GDX files
Keywords: polars,gams,gdx,energy-modeling,optimization
Author: enioarz
License-Expression: MIT AND Zlib
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/enioarz/polars-gdx/releases
Project-URL: Homepage, https://github.com/enioarz/polars-gdx
Project-URL: Issues, https://github.com/enioarz/polars-gdx/issues
Project-URL: Repository, https://github.com/enioarz/polars-gdx

# polars-gdx

Lazy, prefiltered Polars access to [GAMS GDX](https://github.com/GAMS-dev/gdx) files — no GAMS installation required.

- `scan_gdx(path, symbol=...)` returns a `pl.LazyFrame`; nothing is read until `collect()`, and column projection is pushed down via Polars' IO-source interface. `read_gdx(...)` is the eager convenience wrapper (`scan_gdx(...).collect()`, same arguments).
- Predicates on key columns (`key_filter={dim_index: allowed_labels}`, `filter(pl.col(key) == label)` or `filter(pl.col(key).is_in(labels))`) are folded into a **native, index-based prefilter inside the Rust read loop**, so unwanted records are never materialized — unlike the official `gamsapi` reader, which must load everything into pandas first. The prefilter runs inside a bulk C callback (`gdxDataReadRawFastEx`): one FFI crossing for the whole symbol, with early termination for `head(n)`.
- `threads=` (on `scan_gdx`/`read_gdx`) parallelises the raw read across independent file handles. With a first-dimension filter, each worker scans a contiguous range of the first key dimension and whole non-matching ranges are skipped without touching the file. Without one, the data section is split by *file position* at exact record boundaries: a cached restart-position index (one cheap sequential pass, built once per file+symbol) provides the boundaries, and each worker decodes only its own byte range. In both cases results are concatenated in range order so record order matches the serial read exactly, and any boundary/count mismatch falls back to a verified serial read. `threads="auto"` (recommended) uses all cores for symbols of 5M+ records and stays serial for smaller ones. Filters on non-leading dimensions still decode every record within each worker's range, but the decode is now spread across all workers instead of the single-threaded wall-time floor.
- `read_domains(path, symbol=...)` lists the unique labels actually used per index dimension in a single bulk C scan (`gdxGetDomainElements`) — the cost of `unique()` over a key column without reading or materialising any records, valuable on very large symbols.
- Under the hood: hand-written FFI to the vendored MIT-licensed GDX C library, extracted from [lolow/gdxcomp](https://github.com/lolow/gdxcomp), building on [GAMS-dev/gdx](https://github.com/GAMS-dev/gdx).

## Usage

```python
import polars as pl
from polars_gdx import scan_gdx, list_symbols

list_symbols("trnsport.gdx").filter(pl.col("type") == "Param")

# lazy: nothing read yet
x = scan_gdx("trnsport.gdx", symbol="x")

# read only the seattle rows, skipping all others natively
x.filter(pl.col("dim_0") == "seattle").collect()

# eager convenience: read_gdx = scan_gdx(...).collect()
from polars_gdx import read_gdx
df = read_gdx("trnsport.gdx", symbol="x", key_filter={0: ["seattle"]})

# large symbols: parallel raw read across all cores
big = scan_gdx("huge.gdx", symbol="x", threads="auto")

# unique labels per index dimension without reading the records:
from polars_gdx import read_domains
read_domains("trnsport.gdx", symbol="x")
```

### Label matching

Key labels (`key_filter` and `filter(pl.col(key) == label)` predicates) are matched **case-insensitively** (ASCII-folded), the way GAMS does — `"SEATTLE"` matches a stored `"seattle"`. The labels in the produced frame always keep the file's original casing. For predicate filters whose label only differs in case from the stored one, the plugin deliberately skips the native prefilter and re-applies a case-folded predicate on the full read, because Polars' `==` is case-sensitive and would otherwise silently drop the rows.

### File handle lifetime

`scan_gdx` keeps the GDX file open until the returned `LazyFrame` is collected and released (Polars may read the source more than once). Use `read_gdx`, or an explicit `Reader` with its `close()` / context-manager support, to release the handle eagerly — relevant on Windows, where an open handle locks the file.

## Benchmark

vs [`gamsapi`](https://pypi.org/project/gamsapi/) 54.5.0 (pandas-backed), on a 2M-record parameter (80 MB GDX); best of 5 runs. See `benchmarks/bench_vs_gamsapi.py` (run as a CI job on every push).

| Scenario | polars-gdx | gamsapi (pandas) |
| --- | ---: | ---: |
| Full read (2M rows) | 0.128 s | 0.086 s |
| `filter(dim_0 == label)` — 1000 of 2M rows | **0.059 s** | 0.096 s¹ |
| Explicit `key_filter` — 1000 rows | **0.058 s** | — |
| `head(1000)` of 2M rows | **0.001 s** | 0.101 s¹ |

¹ gamsapi has no lazy loading or pushdown: it must materialize the whole symbol, then filter. The advantage of polars-gdx grows with predicate selectivity; `head(n)`/`n_rows` terminates the native read after n records.

## Build

```sh
maturin develop --release   # builds the vendored GDX C library + extension
pytest
```

Requires a C++17 compiler and CMake ≥ 3.5 at build time; runtime has no GAMS dependency.

## Layout

```text
crates/gdx-sys        # raw FFI; builds vendored GDX-dev/gdx via CMake
crates/gdx            # safe RAII wrapper (global lock; lazy UEL interning)
crates/polars-gdx     # PyO3 extension: symbol listing + Arrow streaming with prefiltering
python/polars_gdx     # Python layer: scan_gdx / list_symbols
```

