Metadata-Version: 2.4
Name: rdsframe
Version: 0.4.0a7
Summary: Fast, memory-conscious reader for RDS data.frames
Project-URL: Homepage, https://github.com/mmanaylopez/rdsframe
Project-URL: Repository, https://github.com/mmanaylopez/rdsframe
Project-URL: Changelog, https://github.com/mmanaylopez/rdsframe/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/mmanaylopez/rdsframe/issues
Author: Miguel Antonio Manay López
License-Expression: MIT
License-File: LICENSE
Keywords: dataframe,pandas,parquet,r,rdata,rds
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Provides-Extra: arrow
Requires-Dist: pyarrow>=12; extra == 'arrow'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pandas-stubs>=2.2; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Provides-Extra: parquet
Requires-Dist: duckdb>=1.0; extra == 'parquet'
Requires-Dist: pyarrow>=12; extra == 'parquet'
Description-Content-Type: text/markdown

# rdsframe

`rdsframe` reads common R `data.frame` and `data.table` objects from binary RDS
files directly into pandas. Numeric vectors are filled in their final NumPy
allocation, which avoids the large intermediate Python object tree created by
general-purpose R serialization readers.

> Status: 0.4.0a7 pre-release. Validate results against R for critical pipelines.
> Unsupported R structures fail explicitly; they are never silently coerced.

## Features

- R serialization versions 2 and 3, XDR and native binary formats.
- Uncompressed, gzip, bzip2, and xz containers.
- Integer, double, logical, character, factor (ordered and unordered), `Date`,
  `POSIXct`, `POSIXlt`, `difftime`, raw, and the ALTREP representations R
  actually serializes (compact integer/real sequences, deferred `as.character`
  strings, `sort()`/attribute wrappers). Complex columns become complex pandas
  series or `STRUCT(real, imag)` in Parquet.
- Batched string parsing: STRSXP columns are decoded (or structurally
  skipped) from large chunks instead of three stream reads per element,
  which is what makes text-heavy catalogs practical (see `BENCHMARKS.md`).
- Explicit policies for time zones, invalid timestamps, and heterogeneous
  list-columns; lossy conversion is never activated implicitly.
- A single `data.frame` or a named list of `data.frame` objects -- and, via
  `read_r_object()` or `rdsframe dump`, any other supported R value: plain
  lists, nested structures, matrices, S4 objects (as a dict of slots), and
  environments (as a dict of their contents).
- Reads from a path, raw `bytes`, or any seekable binary stream.
- Character row names become the pandas index; R's compact default row
  numbering stays a `RangeIndex`.
- Nullable pandas integer/boolean dtypes and categorical factors.
- Column-staged Parquet export through DuckDB and a command-line interface.
- Configurable defensive limits for untrusted or unexpectedly large inputs.
- Low-allocation table catalogs and selective extraction by index, name, or
  (for `read_rds()`) column, plus `materialize_uncompressed()` for repeated
  seek-based access to large compressed files.
- Validated against a golden-file corpus written by R 4.5.0 itself
  (`tests/data/r450`, generated by `tests/data/gen_fixtures.R`).

RData workspaces, ASCII serialization, closures/language objects, and custom
third-party ALTREP classes are intentionally unsupported -- and fail with an
error that names the R type, so applications can tell users exactly why a
slower general-purpose fallback is being used.

## Install

```bash
pip install rdsframe
```

For Parquet export:

```bash
pip install "rdsframe[parquet]"
```

## Python API

```python
from rdsframe import read_rds

data = read_rds("measurements.rds")
print(data.head())
```

A named R list returns `dict[str, pandas.DataFrame]`. To require the legacy
first-table behavior, use `read_rds_dataframe()`.

For text-heavy data that still needs to become a pandas DataFrame, keep strings
in Arrow-backed pandas arrays:

```python
data = read_rds("measurements.rds", strings="pyarrow")
```

To read only a few fields from a wide single data.frame, select columns by
zero-based index or exact name. Unselected columns are structurally skipped:
they are never allocated as NumPy arrays or pandas objects.

```python
data = read_rds("measurements.rds", columns=["station", "value", "date"])
```

`columns` requires the RDS root to be a single data.frame. For a multi-table
RDS, select the table first with `extract_rds_tables()` or `to_parquet()`.
Selecting by name performs one extra structural pass to resolve names to
positions (bounded memory, no payload allocation); integer indices skip that
pass entirely.

The saving depends on the type of the skipped columns, not just their count.
RDS stores columns sequentially with variable-length encoding: skipping a
numeric, logical, or raw column in an uncompressed file is a real `seek()`
(near-zero cost); skipping a character column still visits every row to
find where it ends, the same traversal a full read does, so it mainly saves
the decode/interning/pandas-object cost rather than the row scan itself.
`columns` is most valuable for RAM (skipped columns are never allocated) and
for wide tables with many unwanted numeric fields.

## Reading non-tabular RDS files

Not every RDS file is a data.frame. `read_rds()` raises `UnsupportedRDS` for
those on purpose, since silently coercing a plain list into a table would be
worse than failing loudly. Use `read_r_object()` instead:

```python
from rdsframe import read_r_object

data = read_r_object("workspace.rds")
```

A named R list becomes a `dict`; an unnamed list becomes a `list`; a nested
`data.frame` anywhere inside either still becomes a pandas `DataFrame`.
Atomic vectors get the same type rules as a data.frame column (factor,
`Date`, `POSIXct`, `difftime`, matrices via a `dim` attribute) but come back
as a plain Python scalar or list rather than a pandas Series. This is a
general, exploratory reader, not the fast path: it materializes the whole
structure in memory rather than streaming it, so prefer `read_rds()` /
`to_parquet()` whenever the file actually is tabular.

```python
from rdsframe import ReaderLimits, read_rds, to_parquet

limits = ReaderLimits(max_vector_length=100_000_000)
data = read_rds("input.rds", limits=limits, strings="string")

tables = to_parquet("input.rds", "output", compression="zstd")
```

Safety and fidelity policies are explicit:

```python
tables = to_parquet(
    "input.rds",
    "output",
    max_tables=100,            # fail before writing partial results
    max_root_items=10_000,
    posixct_mode="preserve",   # or "utc_naive" for constrained executables
    invalid_timestamp="error", # or "null" to opt into coercion
    list_column_mode="infer",  # or explicit "json" / "string"
)
```

`max_tables` never truncates a file. Exceeding a configured limit raises
`RDSLimitError`, cleans temporary data, and leaves no apparently successful
partial output.

The staging policy is adaptive. By default, a temporary Parquet contains at
most 16 columns or approximately 128 MiB of Arrow buffers, whichever is reached
first. This reduces temporary-file count on wide tables while bounding memory:

```python
tables = to_parquet(
    "input.rds",
    "output",
    stage_max_columns=16,
    stage_max_bytes=128 * 1024 * 1024,
    gc_collect_every=16,  # use 0 to disable explicit cyclic GC
)
```

## CLI

```bash
rdsframe inspect input.rds
rdsframe list input.rds --catalog input.rdsframe.json
rdsframe convert input.rds output/ \
  --table-name measurements \
  --catalog input.rdsframe.json \
  --max-tables 100 \
  --list-column-mode json \
  --invalid-timestamp null
rdsframe dump input.rds
```

Run `rdsframe convert --help` for staging, memory, time-zone and limit controls.

`rdsframe dump` prints **any** supported R object -- not only data.frames --
as an indented tree, so an unknown RDS file can be explored before deciding
how to convert it. Deeply nested structures are summarized rather than
exploded: `--max-items` bounds children per node (default 10), `--max-depth`
bounds nesting (default 8), and `--json` emits the complete object as JSON
instead (data.frames appear as `{"$r_type": "data.frame", ...}` objects).

```text
$ rdsframe dump results.rds --max-items 2 --max-depth 3
<list, 10 items>
  [0] <list, 10 items>
    [0] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    [1] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    ... (+8 more items)
  [1] <list, 10 items>
    ...
```

## List and selectively extract tables

Discover tables without building their pandas, NumPy, Arrow, or Parquet payloads:

```python
from rdsframe import list_rds_tables

catalog = list_rds_tables("workspace.rds")
for table in catalog.tables:
    print(table.index, table.name, table.rows, table.columns)

catalog.save("workspace.rdsframe.json")
```

Extract by zero-based index in one pass:

```python
from rdsframe import extract_rds_tables

extract_rds_tables("workspace.rds", "output", [0, 2, 7])
```

Or select by name while reusing the validated catalog:

```python
from rdsframe import RDSCatalog, extract_rds_tables

catalog = RDSCatalog.load("workspace.rdsframe.json")
extract_rds_tables(
    "workspace.rds",
    "output",
    ["measurements", "stations"],
    catalog=catalog,
)
```

The catalog stores the absolute path, file size and nanosecond modification
time. A stale or foreign catalog raises `RDSCatalogError` before conversion.

RDS is sequential and normally stores list names after all elements. Therefore:

- listing avoids column allocations and temporary outputs, but compressed input
  must still be traversed and decompressed;
- uncompressed binary RDS payloads are skipped with direct seeks; compressed
  payloads are discarded through a reusable bounded buffer;
- integer selection without a catalog converts only chosen tables but may scan
  the remaining stream to recover original names;
- a validated catalog enables early stop after the last selected table and is
  the fastest repeated workflow;
- name selection without a catalog performs one structural listing pass and a
  second selective conversion pass.

The source distribution includes `BENCHMARKS.md` with preliminary reproducible
results and their limitations.

## Memory model

`read_rds()` returns pandas objects and therefore requires the resulting table to
fit in memory. Numeric columns avoid a full-size temporary bytes buffer.

`to_parquet()` is the large-file path: it constructs Arrow-native columns and
stages memory-bounded groups of columns before DuckDB combines them with a
positional join. Character vectors are built from offsets, UTF-8 data and
validity buffers without retaining a Python `str` object for every row. Peak
parser memory is tied to the current column plus the configured staging batch,
not to the entire data frame. DuckDB can spill merge work to disk, controlled by
`memory_limit` and `temp_directory`.

For the lowest possible peak, set `stage_max_columns=1`. For faster conversion
of wide, narrow-column tables, increase the batch limits after benchmarking.

The result is directly queryable without loading it in memory:

```sql
SELECT category, avg(value)
FROM read_parquet('output/measurements.parquet')
WHERE date >= DATE '2025-01-01'
GROUP BY category;
```

## Development

```bash
python -m pip install -e ".[dev,parquet]"
pytest
ruff check .
python -m build
twine check dist/*
```

Before a public release, run the test matrix and publish to TestPyPI first.
The project lives at <https://github.com/mmanaylopez/rdsframe>.

## RData and fallback integration

`rdsframe` deliberately rejects `.RData`/`.rda` workspaces. Applications that
need both formats should inspect the container and use `rdsframe` for supported
binary RDS files, retaining `rdata` or another general reader as fallback for
RData, unsupported ALTREP classes, and richer R objects. An
`UnsupportedRDS` exception is the explicit signal to activate that fallback.

## Conversion policies

- `POSIXct` preserves its R time-zone attribute by default. Applications that
  cannot ship time-zone support may choose `posixct_mode="utc_naive"`; the
  underlying UTC instant is retained and only the display-zone metadata is
  removed.
- `NaN`/R missing timestamps become null. Infinite or out-of-range timestamps
  raise by default; `invalid_timestamp="null"` is an explicit coercion.
- Homogeneous list-columns use Arrow inference. Heterogeneous values raise by
  default. `list_column_mode="json"` produces deterministic UTF-8 JSON, including
  tagged encodings for binary and non-finite values; `"string"` is available
  only when human-readable representation is preferred over structure.
- Ordered factors (`factor(x, ordered = TRUE)`) become an ordered
  `pandas.Categorical`; plain factors are unordered. `to_parquet()` marks the
  underlying Arrow dictionary as ordered too, though the DuckDB staging step
  currently re-materializes any dictionary-encoded column as a plain string
  in the final Parquet file (true for ordinary factors as well, not specific
  to ordering) -- the values are correct, the categorical/dictionary typing
  itself does not yet survive that step.
- `difftime` columns become a pandas `timedelta64` Series (Arrow
  `duration("us")` in Parquet), converted according to the R `units`
  attribute (`secs`, `mins`, `hours`, `days`, or `weeks`).
- A data.frame column with a `dim` attribute (an R matrix or array stored as
  one column) is explicitly unsupported and raises `UnsupportedRDS` rather
  than being silently reshaped or misread.

## Known limitations

- `Rle`/other Bioconductor S4 types, `sf` geometry columns, and
  `data.frame`/matrix-valued columns are not specifically recognized; they
  raise a clear `UnsupportedRDS`/`InvalidRDS`, never a silently wrong result.
- `POSIXlt` values are reconstructed from their wall-clock components
  (`year`/`mon`/`mday`/`hour`/`min`/`sec`) as timezone-naive timestamps; the
  `zone`/`gmtoff` components are not mapped onto pandas time zones because R
  zone abbreviations do not reliably resolve to IANA names.
- ALTREP support covers what R itself serializes with custom state: compact
  integer/real sequences, deferred `as.character` strings, and the `wrap_*`
  wrapper classes (including their attribute slot, so a sorted factor keeps
  its levels). Third-party ALTREP classes with their own serialization
  still raise `UnsupportedRDS`.
- The "native" (non-XDR) binary RDS format has no self-describing byte
  order. `rdsframe` validates the header under the reading machine's order
  and retries the opposite order before giving up, so a cross-endian file
  produces a clear error or a correct read -- but XDR (`saveRDS`'s default)
  remains the only portable choice.
- A CHARSXP with no explicit UTF-8/ASCII/latin-1 flag uses the encoding the
  RDS version-3 header itself declares when it is recognized, else UTF-8.
  For older files where that guess is wrong, pass
  `encoding="windows-1252"` (or any Python codec) to the read functions or
  `--encoding` on the CLI. The same override also guards against files that
  *claim* UTF-8 in their flags but contain differently-encoded bytes: when
  `encoding=` is given, flagged-UTF-8 strings are validated and re-decoded
  with the override on failure. Without it, flags are trusted as-is -- that
  zero-validation fast path is deliberate.
- Parquet has no complex-number type, so complex columns are written as
  `STRUCT(real DOUBLE, imag DOUBLE)`: a lossless *representation*, not a
  native semantic type. Consumers must reassemble `real + imag*i` themselves.
- `difftime` values are normalized to Arrow `duration("us")` in Parquet. The
  elapsed time is exact, but the original display unit (`"weeks"` vs
  `"days"` vs `"secs"`) is not distinguishable from the Parquet type alone.
- Heterogeneous list-columns raise `UnsupportedRDS` unless
  `list_column_mode="json"` or `"string"` is passed. This is deliberate:
  silently coercing mixed structures would violate the no-implicit-loss
  contract, so the caller must pick the representation.
- Environments are read as a plain dict of their contents; the parent scope
  (enclosure) and lock flag are not represented. S4 objects are read as a
  dict of their slots with the class recorded under `"$r_class"`.

## License

MIT © 2026 Miguel Antonio Manay López.
