Metadata-Version: 2.4
Name: rdsframe
Version: 0.4.0b2
Summary: Fast, memory-conscious reader for RDS data.frames
Author: Miguel Antonio Manay López
License-Expression: MIT
Project-URL: Homepage, https://github.com/mmanaylopez/rdsframe
Project-URL: Repository, https://github.com/mmanaylopez/rdsframe
Project-URL: Changelog, https://github.com/mmanaylopez/rdsframe/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/mmanaylopez/rdsframe/issues
Keywords: rds,rdata,pandas,dataframe,r,parquet
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Provides-Extra: arrow
Requires-Dist: pyarrow>=12; extra == "arrow"
Provides-Extra: parquet
Requires-Dist: pyarrow>=12; extra == "parquet"
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.0; extra == "duckdb"
Requires-Dist: pyarrow>=12; extra == "duckdb"
Provides-Extra: polars
Requires-Dist: polars>=1.0; extra == "polars"
Requires-Dist: pyarrow>=12; extra == "polars"
Provides-Extra: zstd
Requires-Dist: zstandard>=0.16; extra == "zstd"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: cython>=3.0; extra == "dev"
Requires-Dist: hypothesis>=6.100; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pandas-stubs>=2.2; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# rdsframe

[![PyPI version](https://img.shields.io/pypi/v/rdsframe?include_prereleases)](https://pypi.org/project/rdsframe/)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/rdsframe/)
[![CI](https://github.com/mmanaylopez/rdsframe/actions/workflows/ci.yml/badge.svg)](https://github.com/mmanaylopez/rdsframe/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

`rdsframe` reads common R `data.frame` and `data.table` objects from binary RDS
files directly into pandas. Numeric vectors are filled in their final NumPy
allocation, which avoids the large intermediate Python object tree created by
general-purpose R serialization readers.

> Status: 0.4.0b2 pre-release (beta). Validate results against R for critical pipelines.
> Unsupported R structures fail explicitly; they are never silently coerced.

## Features

- R serialization versions 2 and 3, XDR and native binary formats.
- Uncompressed, gzip, bzip2, xz, and zstd containers (zstd is what
  R >= 4.5 writes for `compress = "zstd"`; reading it needs Python >= 3.14
  or the `rdsframe[zstd]` extra).
- Integer, double, logical, character, factor (ordered and unordered), `Date`,
  `POSIXct`, `POSIXlt`, `difftime`, raw, and the ALTREP representations R
  actually serializes (compact integer/real sequences, deferred `as.character`
  strings, `sort()`/attribute wrappers). Complex columns become complex pandas
  series or `STRUCT(real, imag)` in Parquet.
- Batched string parsing: STRSXP columns are decoded (or structurally
  skipped) from large chunks instead of three stream reads per element,
  which is what makes text-heavy catalogs practical (see `BENCHMARKS.md`).
- Optional compiled scanner: Cython accelerates only structural CHARSXP
  skipping when the extension is available; materializing strings remains on
  the tested Python path, as does all parsing without the module.
- Explicit policies for time zones, invalid timestamps, and heterogeneous
  list-columns; lossy conversion is never activated implicitly.
- A single `data.frame` or a named list of `data.frame` objects -- and, via
  `read_r_object()` or `rdsframe dump`, any other supported R value: plain
  lists, nested structures, matrices, S4 objects (as a dict of slots), and
  environments (as a dict of their contents).
- Reads from a path, raw `bytes`, or any seekable binary stream.
- Character row names become the pandas index; R's compact default row
  numbering stays a `RangeIndex`.
- Nullable pandas integer/boolean dtypes and categorical factors.
- Public Arrow-table reads plus Parquet export through PyArrow or the optional
  memory-bounded DuckDB engine, and a command-line interface.
- Deferred `open_rds()` datasets with structural schemas and column projection,
  exact/metadata-only inspection modes, and Arrow-backed Polars/DuckDB adapters.
- Configurable defensive limits for untrusted or unexpectedly large inputs.
- Low-allocation table catalogs and selective extraction by index, name, or
  (for `read_rds()`) column, plus `materialize_uncompressed()` for repeated
  seek-based access to large compressed files.
- Validated against a golden-file corpus written by R 4.5.0 itself
  (`tests/data/r450`, generated by `tests/data/gen_fixtures.R`).

RData workspaces, ASCII serialization, closures/language objects, and custom
third-party ALTREP classes are intentionally unsupported -- and fail with an
error that names the R type, so applications can tell users exactly why a
slower general-purpose fallback is being used.

## How it compares

One fresh process per read on the same machine (Windows 11, Python 3.12,
2026-07-16), wall time / peak RSS, against `pyreadr 0.5.6` (librdata, C) and
`rdata 1.1.0`. Synthetic inputs are uncompressed; details, versions, and
caveats in `BENCHMARKS.md` -- treat these as one data point, not a universal
claim.

| Input | rdsframe | pyreadr | rdata |
| --- | ---: | ---: | ---: |
| 2M rows x 8 numeric columns (88 MiB) | **0.7 s / 199 MB** | 1.8 s / 469 MB | 1.3 s / 490 MB |
| 1M rows x 5 text-heavy columns | **2.6 s / 228 MB** | 16.9 s / 418 MB | 39.3 s / 1.7 GB |
| nflverse play-by-play 2023 (49,665 x 372, gzip) | **4.3 s / 322 MB** | 4.7 s / 766 MB | 94 s / 3.7 GB |
| 123 MiB gzip, root = named list of 6 data.frames | **32 s (all tables)** | returns `{}` silently | impractical |
| Catalog only (`list_rds_tables`) of that same file | **3.9 s** | n/a | n/a |

Two structural differences matter more than the timings. pyreadr (librdata)
cannot read an RDS whose root is a *list* of data.frames -- it returns an
empty dict without raising -- while that layout is exactly what `rdsframe`
targets, including selective extraction. And where all three readers can read
the same file, `rdsframe`'s output was verified value-for-value against
pyreadr (372/372 columns identical on the nflverse file) and against R itself
via checksums.

## Install

```bash
pip install rdsframe
```

For Parquet export:

```bash
pip install "rdsframe[parquet]"
```

This installs PyArrow and does not require DuckDB. Install the streaming,
column-staged engine when conversion must keep peak memory tied to a column
batch rather than a complete table:

```bash
pip install "rdsframe[duckdb]"
```

For Polars:

```bash
pip install "rdsframe[polars]"
```

For zstd-compressed RDS on Python < 3.14:

```bash
pip install "rdsframe[zstd]"
```

Source builds try to compile the small Cython-generated scanner from portable
C, but compilation failure is optional and never prevents installation. Check
the active installation with `rdsframe.compiled_backend_available()`.

## Python API

```python
from rdsframe import read_rds

data = read_rds("measurements.rds")
print(data.head())
```

A named R list returns `dict[str, pandas.DataFrame]`. To require the legacy
first-table behavior, use `read_rds_dataframe()`.

For text-heavy data that still needs to become a pandas DataFrame, keep strings
in Arrow-backed pandas arrays:

```python
data = read_rds("measurements.rds", strings="pyarrow")
```

To bypass pandas entirely, request public Arrow tables:

```python
from rdsframe import read_rds_arrow

table = read_rds_arrow("measurements.rds")
```

A named list returns `dict[str, pyarrow.Table]`.

### Deferred datasets and inspection

`open_rds()` creates a lightweight handle. Accessing `schema` performs or
reuses the structural catalog scan; payloads are not converted to NumPy,
pandas, Arrow or Polars until a terminal operation is requested:

```python
from rdsframe import open_rds

dataset = open_rds("measurements.rds")

print(dataset.schema)
print(dataset.columns)
print(dataset.shape)

preview = dataset[["peso", "edad"]].head(10)
peso = dataset["peso"].collect()
arrow = dataset.select(["peso", "edad"]).to_arrow()
```

For a named list of data.frames, choose one explicitly:

```python
stations = open_rds("workspace.rds").table("stations")
```

Projection is lazy and a single-root data.frame skips unselected columns.
`head()` currently limits the returned pandas result, not the underlying RDS
vector read: R serializes complete columns, so the selected columns are still
consumed. Compressed input must also be decompressed sequentially to reach a
later column.

Metadata-only inspection never materializes column payloads:

```python
from rdsframe import inspect_rds

info = inspect_rds("measurements.rds")
print(info.rows, info.columns, info.compression)
for column in info.tables[0].schema:
    print(column.name, column.logical_type, column.factor, column.estimated_bytes)
```

Fixed-width columns have a structural memory estimate. Text/list memory and
missing counts cannot be known without reading their elements; request the
explicit scan when exact Arrow buffer sizes and null counts are needed:

```python
info = inspect_rds("measurements.rds", mode="scan")
print(info.statistics_complete)
print(info.tables[0].schema[0].missing_count)
```

### Polars and DuckDB

Both adapters reuse the Arrow conversion and avoid pandas:

```python
from rdsframe import open_rds, read_rds_polars

frame = read_rds_polars("measurements.rds")
frame = open_rds("measurements.rds").select(["edad", "peso"]).to_polars()
```

DuckDB can consume the projected table as a relation or registered SQL view:

```python
dataset = open_rds("measurements.rds").select(["sexo", "edad", "peso"])

relation = dataset.to_duckdb()
result = relation.filter("edad >= 18").aggregate("sexo, avg(peso)")

connection = dataset.register_duckdb("measurements")
result = connection.sql("""
    SELECT sexo, avg(peso)
    FROM measurements
    WHERE edad >= 18
    GROUP BY sexo
""")
```

This is a Python/Arrow registration, not yet a native DuckDB
`read_rds('path')` table function; query projection should therefore be applied
on the `RDSDataset` before registration.

To read only a few fields from a wide single data.frame, select columns by
zero-based index or exact name. Unselected columns are structurally skipped:
they are never allocated as NumPy arrays or pandas objects.

```python
data = read_rds("measurements.rds", columns=["station", "value", "date"])
```

`columns` requires the RDS root to be a single data.frame. For a multi-table
RDS, select the table first with `extract_rds_tables()` or `to_parquet()`.
Selecting by name performs one extra structural pass to resolve names to
positions (bounded memory, no payload allocation); integer indices skip that
pass entirely.

The saving depends on the type of the skipped columns, not just their count.
RDS stores columns sequentially with variable-length encoding: skipping a
numeric, logical, or raw column in an uncompressed file is a real `seek()`
(near-zero cost); skipping a character column still visits every row to
find where it ends, the same traversal a full read does, so it mainly saves
the decode/interning/pandas-object cost rather than the row scan itself.
`columns` is most valuable for RAM (skipped columns are never allocated) and
for wide tables with many unwanted numeric fields.

## Reading non-tabular RDS files

Not every RDS file is a data.frame. `read_rds()` raises `UnsupportedRDS` for
those on purpose, since silently coercing a plain list into a table would be
worse than failing loudly. Use `read_r_object()` instead:

```python
from rdsframe import read_r_object

data = read_r_object("workspace.rds")
```

A named R list becomes a `dict`; an unnamed list becomes a `list`; a nested
`data.frame` anywhere inside either still becomes a pandas `DataFrame`.
Atomic vectors get the same type rules as a data.frame column (factor,
`Date`, `POSIXct`, `difftime`, matrices via a `dim` attribute) but come back
as a plain Python scalar or list rather than a pandas Series. A *named*
atomic vector (`c(a = 1, b = 2)`) becomes a `dict` like a named list does,
and a class-less matrix keeps its native NumPy dtype (`dimnames` are not
represented). This is a
general, exploratory reader, not the fast path: it materializes the whole
structure in memory rather than streaming it, so prefer `read_rds()` /
`to_parquet()` whenever the file actually is tabular.

```python
from rdsframe import ReaderLimits, read_rds, to_parquet

limits = ReaderLimits(max_vector_length=100_000_000)
data = read_rds("input.rds", limits=limits, strings="string")

tables = to_parquet("input.rds", "output", compression="zstd")
```

`engine="auto"` selects DuckDB when installed and otherwise writes with
PyArrow. Use `engine="pyarrow"` to require the dependency-light route or
`engine="duckdb"` to require column-staged, spill-to-disk conversion.

Safety and fidelity policies are explicit:

```python
tables = to_parquet(
    "input.rds",
    "output",
    max_tables=100,            # fail before writing partial results
    max_root_items=10_000,
    posixct_mode="preserve",   # or "utc_naive" for constrained executables
    invalid_timestamp="error", # or "null" to opt into coercion
    list_column_mode="infer",  # or explicit "json" / "string"
)
```

`max_tables` never truncates a file. Exceeding a configured limit raises
`RDSLimitError`, cleans temporary data, and leaves no apparently successful
partial output.

The staging policy is adaptive. By default, a temporary Parquet contains at
most 16 columns or approximately 128 MiB of Arrow buffers, whichever is reached
first. This reduces temporary-file count on wide tables while bounding memory:

```python
tables = to_parquet(
    "input.rds",
    "output",
    stage_max_columns=16,
    stage_max_bytes=128 * 1024 * 1024,
    gc_collect_every=16,  # use 0 to disable explicit cyclic GC
)
```

## CLI

```bash
rdsframe inspect input.rds
rdsframe list input.rds --cache
rdsframe list input.rds --catalog input.rdsframe.json
rdsframe convert input.rds output/ \
  --engine pyarrow \
  --table-name measurements \
  --catalog input.rdsframe.json \
  --max-tables 100 \
  --list-column-mode json \
  --invalid-timestamp null
rdsframe dump input.rds
```

Run `rdsframe convert --help` for staging, memory, time-zone and limit controls.

`rdsframe dump` prints **any** supported R object -- not only data.frames --
as an indented tree, so an unknown RDS file can be explored before deciding
how to convert it. Deeply nested structures are summarized rather than
exploded: `--max-items` bounds children per node (default 10), `--max-depth`
bounds nesting (default 8), and `--json` emits the complete object as JSON
instead (data.frames appear as `{"$r_type": "data.frame", ...}` objects).

```text
$ rdsframe dump results.rds --max-items 2 --max-depth 3
<list, 10 items>
  [0] <list, 10 items>
    [0] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    [1] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    ... (+8 more items)
  [1] <list, 10 items>
    ...
```

## List and selectively extract tables

Discover tables without building their pandas, NumPy, Arrow, or Parquet payloads:

```python
from rdsframe import list_rds_tables

catalog = list_rds_tables("workspace.rds", cache=True)
for table in catalog.tables:
    print(table.index, table.name, table.rows, table.columns)
```

With `cache=True`, the first scan writes `workspace.rdsframe.json`; later calls
return it immediately while source path, size, and modification time match.
Pass an explicit path as `cache="catalogs/workspace.json"` when desired.

Extract by zero-based index in one pass:

```python
from rdsframe import extract_rds_tables

extract_rds_tables("workspace.rds", "output", [0, 2, 7])
```

Or select by name while reusing the validated catalog:

```python
from rdsframe import RDSCatalog, extract_rds_tables

catalog = RDSCatalog.load("workspace.rdsframe.json")
extract_rds_tables(
    "workspace.rds",
    "output",
    ["measurements", "stations"],
    catalog=catalog,
)
```

The catalog stores the absolute path, file size and nanosecond modification
time. An explicitly loaded stale or foreign catalog raises `RDSCatalogError`
before conversion; the opt-in automatic cache rebuilds stale sidecars.
Name-based `to_parquet()` / `extract_rds_tables()` selection automatically
creates and reuses this sidecar when no explicit catalog is supplied.

RDS is sequential and normally stores list names after all elements. Therefore:

- listing avoids column allocations and temporary outputs, but compressed input
  must still be traversed and decompressed;
- uncompressed binary RDS payloads are skipped with direct seeks; compressed
  payloads are discarded through a reusable bounded buffer;
- integer selection without a catalog converts only chosen tables but may scan
  the remaining stream to recover original names;
- a validated catalog enables early stop after the last selected table and is
  the fastest repeated workflow;
- the first name selection without an explicit catalog performs a structural
  listing pass and saves it; subsequent selections reuse the validated cache.

The source distribution includes `BENCHMARKS.md` with preliminary reproducible
results and their limitations.

## Memory model

`read_rds()` returns pandas objects and therefore requires the resulting table to
fit in memory. Numeric columns avoid a full-size temporary bytes buffer.

`to_parquet(engine="duckdb")` is the lowest-memory path: it constructs
Arrow-native columns and stages bounded groups before DuckDB combines them with
a positional join. Character vectors are built from offsets, UTF-8 data and
validity buffers without retaining a Python `str` object for every row. Peak
parser memory is tied to the current column plus the configured staging batch,
not to the entire data frame. DuckDB can spill merge work to disk, controlled by
`memory_limit` and `temp_directory`. The PyArrow engine and `read_rds_arrow()`
materialize a complete table, trading higher peak memory for fewer dependencies.

For the lowest possible peak, set `stage_max_columns=1`. For faster conversion
of wide, narrow-column tables, increase the batch limits after benchmarking.

The result is directly queryable without loading it in memory:

```sql
SELECT category, avg(value)
FROM read_parquet('output/measurements.parquet')
WHERE date >= DATE '2025-01-01'
GROUP BY category;
```

## Development

```bash
python -m pip install -e ".[dev,duckdb,polars,zstd]"
pytest
ruff check .
mypy src/rdsframe
pytest --cov=rdsframe --cov-fail-under=80
python -m build
twine check dist/*
```

Before a public release, run the test matrix and publish to TestPyPI first.
The project lives at <https://github.com/mmanaylopez/rdsframe>.

### Releasing

1. Bump the version in `pyproject.toml` and `src/rdsframe/__init__.py`, and
   date the `CHANGELOG.md` section.
2. Commit, push, and wait for the full CI matrix to pass (it validates the
   Cython build on the three platforms, the pure-Python fallback, and the
   minimum supported NumPy/pandas versions -- environments a single dev
   machine cannot cover).
3. Tag `vX.Y.Z` and push the tag: `release.yml` builds the sdist plus binary
   wheels via cibuildwheel and publishes through PyPI Trusted Publishing
   (one-time setup: PyPI project settings -> Publishing -> GitHub publisher
   for this repo, workflow `release.yml`, environment `pypi`).
4. Manual fallback: upload **only the sdist** (`python -m build --sdist`,
   then `twine upload dist/*.tar.gz`). Never upload a locally built wheel:
   without a working compiler, setuptools still tags it platform-specific
   even though the accelerator is missing, and that wheel would shadow the
   sdist for every user of that platform.

## RData and fallback integration

`rdsframe` deliberately rejects `.RData`/`.rda` workspaces. Applications that
need both formats should inspect the container and use `rdsframe` for supported
binary RDS files, retaining `rdata` or another general reader as fallback for
RData, unsupported ALTREP classes, and richer R objects. An
`UnsupportedRDS` exception is the explicit signal to activate that fallback.

## Conversion policies

- `POSIXct` preserves its R time-zone attribute by default. Applications that
  cannot ship time-zone support may choose `posixct_mode="utc_naive"`; the
  underlying UTC instant is retained and only the display-zone metadata is
  removed.
- `NaN`/R missing timestamps become null. Infinite or out-of-range timestamps
  raise by default; `invalid_timestamp="null"` is an explicit coercion.
- Homogeneous list-columns use Arrow inference. Heterogeneous values raise by
  default. `list_column_mode="json"` produces deterministic UTF-8 JSON, including
  tagged encodings for binary and non-finite values; `"string"` is available
  only when human-readable representation is preferred over structure.
- A factor with an explicit NA level (`addNA()`) maps values at that level
  to missing: pandas/Arrow dictionaries cannot hold a null category, and
  mapping it to `""` would collide with a genuine empty-string level.
- Ordered factors (`factor(x, ordered = TRUE)`) become an ordered
  `pandas.Categorical`; plain factors are unordered. `to_parquet()` marks the
  underlying Arrow dictionary as ordered too, though the DuckDB staging step
  currently re-materializes any dictionary-encoded column as a plain string
  in the final Parquet file (true for ordinary factors as well, not specific
  to ordering) -- the values are correct, the categorical/dictionary typing
  itself does not yet survive that step.
- `difftime` columns become a pandas `timedelta64` Series (Arrow
  `duration("us")` in Parquet), converted according to the R `units`
  attribute (`secs`, `mins`, `hours`, `days`, or `weeks`).
- A data.frame column with a `dim` attribute (an R matrix or array stored as
  one column) is explicitly unsupported and raises `UnsupportedRDS` rather
  than being silently reshaped or misread.

## Known limitations

- `Rle`/other Bioconductor S4 types, `sf` geometry columns, and
  `data.frame`/matrix-valued columns are not specifically recognized; they
  raise a clear `UnsupportedRDS`/`InvalidRDS`, never a silently wrong result.
- `POSIXlt` values are reconstructed from their wall-clock components
  (`year`/`mon`/`mday`/`hour`/`min`/`sec`) as timezone-naive timestamps; the
  `zone`/`gmtoff` components are not mapped onto pandas time zones because R
  zone abbreviations do not reliably resolve to IANA names.
- ALTREP support covers what R itself serializes with custom state: compact
  integer/real sequences, deferred `as.character` strings, and the `wrap_*`
  wrapper classes (including their attribute slot, so a sorted factor keeps
  its levels). Third-party ALTREP classes with their own serialization
  still raise `UnsupportedRDS`.
- The "native" (non-XDR) binary RDS format has no self-describing byte
  order. `rdsframe` validates the header under the reading machine's order
  and retries the opposite order before giving up, so a cross-endian file
  produces a clear error or a correct read -- but XDR (`saveRDS`'s default)
  remains the only portable choice.
- A CHARSXP with no explicit UTF-8/ASCII/latin-1 flag uses the encoding the
  RDS version-3 header itself declares when it is recognized, else UTF-8.
  For older files where that guess is wrong, pass
  `encoding="windows-1252"` (or any Python codec) to the read functions or
  `--encoding` on the CLI. The same override also guards against files that
  *claim* UTF-8 in their flags but contain differently-encoded bytes: when
  `encoding=` is given, flagged-UTF-8 strings are validated and re-decoded
  with the override on failure. Without it, flags are trusted as-is -- that
  zero-validation fast path is deliberate.
- Parquet has no complex-number type, so complex columns are written as
  `STRUCT(real DOUBLE, imag DOUBLE)`: a lossless *representation*, not a
  native semantic type. Consumers must reassemble `real + imag*i` themselves.
- `difftime` values are normalized to Arrow `duration("us")` in Parquet. The
  elapsed time is exact, but the original display unit (`"weeks"` vs
  `"days"` vs `"secs"`) is not distinguishable from the Parquet type alone.
- Heterogeneous list-columns raise `UnsupportedRDS` unless
  `list_column_mode="json"` or `"string"` is passed. This is deliberate:
  silently coercing mixed structures would violate the no-implicit-loss
  contract, so the caller must pick the representation.
- Environments are read as a plain dict of their contents; the parent scope
  (enclosure) and lock flag are not represented. S4 objects are read as a
  dict of their slots with the class recorded under `"$r_class"`.

## License

MIT © 2026 Miguel Antonio Manay López.
