Metadata-Version: 2.5
Name: deltacat-io-core
Version: 0.1.48a1
Summary: Shared local IO execution layer for DeltaCAT read/write clients.
Author: Ray Team
License-Expression: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.13,>=3.10
Requires-Dist: boto3<2,>=1.41.0
Requires-Dist: cryptography<47,>=46
Requires-Dist: fsspec
Requires-Dist: msgpack~=1.0.7
Requires-Dist: numpy<2,>=1.23
Requires-Dist: pyarrow<24,>=17
Requires-Dist: pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<2.10.0,>=2.5
Requires-Dist: typing-extensions>=4.6.1
Provides-Extra: all
Requires-Dist: av==15.0.0; extra == 'all'
Requires-Dist: daft==0.5.22; extra == 'all'
Requires-Dist: fastavro; extra == 'all'
Requires-Dist: pandas==2.2.3; extra == 'all'
Requires-Dist: polars==1.29.0; extra == 'all'
Requires-Dist: pylance>=0.37.0; extra == 'all'
Requires-Dist: pyyaml<7,>=6; extra == 'all'
Requires-Dist: zarr<4,>=3; extra == 'all'
Provides-Extra: daft
Requires-Dist: daft==0.5.22; extra == 'daft'
Provides-Extra: dataloader
Requires-Dist: av==15.0.0; extra == 'dataloader'
Requires-Dist: polars==1.29.0; extra == 'dataloader'
Provides-Extra: io
Requires-Dist: fastavro; extra == 'io'
Provides-Extra: lance
Requires-Dist: pylance>=0.37.0; extra == 'lance'
Provides-Extra: lerobot
Requires-Dist: av==15.0.0; extra == 'lerobot'
Requires-Dist: polars==1.29.0; extra == 'lerobot'
Provides-Extra: packds
Requires-Dist: av==15.0.0; extra == 'packds'
Provides-Extra: pandas
Requires-Dist: pandas==2.2.3; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars==1.29.0; extra == 'polars'
Provides-Extra: umi
Requires-Dist: av==15.0.0; extra == 'umi'
Requires-Dist: polars==1.29.0; extra == 'umi'
Requires-Dist: pyyaml<7,>=6; extra == 'umi'
Requires-Dist: zarr<4,>=3; extra == 'umi'
Description-Content-Type: text/markdown

# deltacat-io-core

`deltacat-io-core` is the shared local execution layer for DeltaCAT reads and
writes.

It is used by both:

- `deltacat-client` for direct thin-plan execution
- `deltacat` for shared local execution and compatibility wrappers

## Naming

- distribution/package name: `deltacat-io-core`
- Python import module: `deltacat_io_core`

The distribution uses dashes for consistency with `deltacat-client`. The import
module keeps underscores because Python module names cannot contain `-`.

## Scope

`deltacat-io-core` owns the code that should behave the same regardless of
whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

- direct execution of thin `Plan` objects
- MOR execution for thin and thick paths
- local file materialization and manifest building
- bounded, authority-bound recovery-frontier index construction and scanning
- schema alignment and table conversion helpers
- sort-aware file ordering and manifest handling
- shared compaction/MOR helper layers and model types
- format-specific local readers/writers

## Non-Goals

`deltacat-io-core` does not own:

- server routes or REST/MCP request handling
- authoritative catalog/storage mutations
- native Ray job orchestration surfaces
- public end-user API shape for `deltacat` or `deltacat-client`

It is a shared implementation layer, not the top-level user product.

For bounded nested-reference enumeration shared by admission callers, see
[source dependency planning](SOURCE_DEPENDENCIES.md). Its PackDS adapter reuses
the native reference decoder and returns exact object/binding metadata without
reading media or acquiring storage capabilities.

## Architecture

The current read architecture is:

1. The server resolves a thin `Plan`.
2. `client.catalog.read(plan=...)` executes that plan directly through
   `deltacat-io-core`.
3. `dc.read_table(plan=...)` for thin plans also executes through the same
   shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan
execution. The plan contract is expected to carry the metadata required for
direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared
   helpers.
2. The authoritative commit still happens through DeltaCAT server/native
   boundaries.
3. Shared write-preparation and manifest logic lives in `deltacat-io-core`.

## Installation

Base install:

```bash
uv pip install deltacat-io-core
```

Optional extras:

- `deltacat-io-core[io]` for local file readers/writers (`pyarrow`, `fastavro`)
- `deltacat-io-core[pandas]` for Pandas conversions
- `deltacat-io-core[polars]` for Polars conversions and lazy scan helpers
- `deltacat-io-core[daft]` for Daft conversions and lazy scan helpers
- `deltacat-io-core[lance]` for Lance dataset support
- `deltacat-io-core[umi]` for aligned-Zarr UMI input and LeRobotV2 video output
- `deltacat-io-core[all]` for the full local IO stack

## Request-owned S3 clients

Storage clients must be created from a complete authority-vetted
`StorageAccessRef`, through `storage_credentials`. The default factory reuses
its authority-scoped cache. Callers that need isolated request hooks can opt in
to `create_boto3_s3_client_for_access_with_retirement_token(...,
request_private=True)`, which bypasses the shared client cache and returns an
owned client and retirement token. The bounded authority snapshot retains all
fields, including credential-source/profile intent; no ambient fallback is
added.

`borrow_private_boto3_s3_client(client, token)` yields the concrete SDK client
and frozen lease facts. It requires the exact live private token in its creator
process and captured explicit credentials with sufficient expiry headroom.
The additional borrow prevents parent-token retirement from closing the client
mid-request. Install hooks before use, retain the scope through response-body
drain, and release the original retirement token when the request owner is done.
Facts are process-local, not a transferable grant. This helper supplies no
request budgets, locality proof, auth-body decoder or automatic replay.

Changed/expired authority holds that capability: leave and drain the scope,
then obtain fresh exact authority outside it within the original deadline.
Transient SDK errors propagate to the caller's existing recovery policy;
neither credentials nor clients refresh inside the concrete borrow. Production
callers remain on their existing route until their request accounting and
authority integration are qualified separately.

### Observing an existing concrete SDK receiver

`run_boto3_s3_read_once` and `run_boto3_s3_mutation_once` accept the optional
`client_observer_scope(actual_client)` context factory. Its default is `None`.
For renewable proxies, it enters after native acquisition and explicit-lease
headroom checks, before model/dispatch guards, and remains active through the
read consumer and body drains. Each acquisition observes its actual current
SDK client, including a replacement generation; attaching to `proxy.meta`
before acquisition is not equivalent. The scope spans existing bounded read
attempts; mutation calls still never replay.

For a caller that already owns a retained HEAD/MPU/readback lifetime,
`borrow_boto3_s3_observed_client(client, operation=...,
client_observer_scope=...)` yields that actual receiver under the proxy's
native active-borrow accounting. Keep it in the existing `ExitStack` until all
bodies have drained. Observer exit precedes release; it cannot suppress or
replace a primary SDK/body error. There is no refresh inside this retained
borrow and no new private token or replacement client. A directly supplied
SDK client remains owned by its original owner and is not closed on scope exit.

This is an observation seam, not authority, private event-hook ownership or a
new retry controller. Callers retain their exact access-ref expiry, deadline,
scope, attempt-budget and concurrency-isolation checks. In particular an
observer on a shared client must isolate its own concurrent operation hooks.
On headroom refusal, leave/drain and obtain fresh authority through the owner
outside the scope. No writer/relay consumer is activated by this helper alone.

### Private read attempt and XML observation

The unused `s3_read_observation.observe_private_s3_read` helper composes the
factory's genuine private-token borrow with temporary SDK hooks. Required
`admit_attempt(operation, prepared_request)` and `check()` callbacks belong to
the request owner: retain exact key/range/IfMatch/version/auth scope, shared
attempt accounting and the original claim/deadline. No callback or policy
default admits requests. Hooks observe each native GetObject/CreateSession
before-send, including retries, redirects and hidden Express authentication.
These are admissions, not proof that network bytes were sent. The helper does
not change SDK retry configuration or activate any read/writer consumer.
An owner callback failure is kept outside the SDK retry classifier and its
identical original exception is restored when leaving the scope. Catch errors
outside that scope; swallowing its internal refusal cannot permit another
admission or turn scope exit into success. Real SDK transport failures still
use the SDK's configured retry policy.
An owner refusal after a response arrives uses that same latch and closes any
unreturned streaming body before releasing decoder workspace. A secondary close
failure adds only a static note to the original refusal; successful bodies are
left untouched for the caller to drain.

An explicit `S3XmlLimits` bounds arrived body bytes, elements, depth,
attributes and UTF-8 text before the real REST-XML parser runs. A SAX Expat
guard produces no credential mapping or DOM and rejects DTD/entity input.
Headers-only errors remain available to native SDK redirect/retry handling.
The raw body has already arrived; this is not a pre-wire allocation or hard
RSS bound. The owner supplies `decoder_workspace()` covering both guard and
SDK parsing for the whole scope. Guard messages contain no body values; this
does not redact independent SDK debug logging or authorize logging credentials.

Keep the scope alive until all response bodies are consumed/closed, with the
same pending-byte/FD owner. It pins the concrete client without refreshing or
replaying mutations. Concurrent observer scopes on that receiver refuse; a
hook-removal failure preserves a primary exception and prevents observer reuse
of that capability. Drain/discard it and acquire fresh exact authority through
the owner within the original deadline. Shared clients are never stamped as
private, and production policy, scoped range transport and caller activation
remain separate integration work.

## Read Capabilities

Selected companion reads can explicitly declare one predecessor physical schema
and a current schema that appends nullable fields. Existing fields, including
their order, nullability, nested types and metadata, must match exactly. Each
Parquet or Lance footer is checked against one of those complete schemas before
projection. Existing arrays are retained unchanged; newly introduced fields are
null for predecessor files. This is separate from catalog schema evolution and
does not authorize a write or rewrite existing files.

The current schema is bound before creating the selected contract and its input
coverage. The same executed plan carries the predecessor reference to downstream
validators. Historical contract hashes may match only the current or declared
predecessor schema; all creator, snapshot and exact input-revision checks remain.
Finite episode filters still run at the file source. Predicates on newly added
columns run over the bounded projected result, where absent fields are null.
Metadata, rows and retained Arrow buffers keep the caller's explicit limits;
these post-decode checks do not promise a bound on the file decoder's peak memory.

The shared read executor currently handles:

- schema-table reads
- schemaless manifest-table reads
- MOR reads
- direct `pyarrow`, `pandas`, `polars`, `numpy`, `daft`, and `ray_dataset`
  outputs where supported
- lazy `pyarrow_parquet`
- lazy `lance`

It also enforces direct validation for unsupported combinations, for example:

- schemaless + `pyarrow_parquet`
- schemaless + `lance`
- mixed-content lazy plans for format-specific readers
- unknown content types in the shared path

### Polars / Daft Capability Matrix

The shared executor applies the same capability decision in thin
`execute_read_plan(...)` execution and in thick reads that delegate into
that shared path.

| Engine | Content | v1 behavior |
| --- | --- | --- |
| Polars | Parquet | Lazy scan via `pl.scan_parquet(...)` when the existing local preconditions hold |
| Polars | Lance | Explicit eager fallback; no reader-level Lance row-filter pushdown |
| Polars | PackDS | Same as Lance; PackDS plans stay on the explicit eager Lance fallback |
| Daft | Parquet | Lazy scan via shared `build_daft_lazy_scan(...)` when the group is local/shared-eligible |
| Daft | Lance | Lazy only for a single dataset on the shared local path; multi-dataset falls back eagerly |
| Daft | PackDS | Same as Lance under PackDS v5: a single pruned episode dataset can use native lazy Lance scanning; multi-episode plans fall back eagerly |

Notes:

- Mixed-schema lazy eligibility on the shared path requires per-file
  `schema_id` lookups plus top-level schema information with resolvable
  field types, whether that comes from `schema_serialized` or a typed
  top-level `schema` summary.
- JSON schema summaries and requests share the canonical datatype codec in
  `schema_types.py`. Readable scalar names follow PyArrow (`float64`, not
  PyArrow's `double` display text); parameter-rich types use a lossless
  one-field Arrow IPC token. Request parsing is strict, while response/plan
  decoding narrowly tolerates the five scalar spellings emitted by older
  DeltaCAT servers so rolling upgrades and durable plans remain readable.
- On the shared Daft path, non-identity Parquet content encodings (for
  example `.parquet.gz`) stay on the eager PyArrow path.
- When the process is pinned to `DAFT_RUNNER=ray`, the shared local Daft
  lazy path declines and falls back to the eager shared path instead of
  spawning a Ray-backed local lazy scan.

## Write Capabilities

The shared write layer currently covers:

- write input normalization
- local data materialization
- manifest construction for existing files and datasets
- schema/read compatibility helpers
- standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries
still remain on the native/server side where they belong.

### Long-running Lance writes

`write_lance_batch_stream` accepts an optional `refresh_access_ref` callback from
an existing scoped credential owner. The shared credential factory checks the
original binding/scope/routing and provider intent on each refresh and emits an
engine refresh deadline before actual expiry. The owner's existing lease cache
keeps vending independent of fragment count. A failed refresh is retained for
that write attempt so the engine cannot amplify it into repeated credential
requests; job-level retry supplies a fresh stream and output path.

The writer uses Lance's public namespace interface only as a fixed-path
credential adapter (`namespace` on pylance3, `namespace_client` on pylance11).
It does not create catalog
namespaces, implement a second writer, or retry/delete partially consumed
streams. Unsupported writers refuse before stream consumption. Nonrenewing and
local callers retain the ordinary write path.

`open_prepared_lance_dataset` accepts the same scoped `access_ref` and optional
`refresh_access_ref` pair for native in-flight reads. It supplies initial routing
options as well as the native provider: the callback alone is insufficient for
Lance's bootstrap. Pylance11 uses the same fixed-path namespace adapter for
reads instead of3's direct provider argument. Both generated-model bootstrap
requests and mapping-shaped native refresh requests validate the fixed ID/path.
Unsupported reader interfaces refuse before opening storage. Earlier source
qualification described the3.0.0 environment as11.0.0 incorrectly; actual engine
versions and installed wheel qualification must be recorded independently.

`storage_read_support.EntryReadAuthority` reuses `RenewingStorageAuthority` for
one job's manifest reads. Path/revision validation precedes lease reuse; at most
128 scoped bindings are retained. The owner keeps an independent copy of each
original binding policy and refuses conflicting policies for that same scope.
Credential refresh uses the original policy, not a mutable current revision.
Fresh reads share one acquisition per binding/scope; a failed acquisition is
retained for the attempt, including concurrent native callbacks. No catalog RPC
or global credential cache is introduced. Training inventory, native payload
and media copies share that owner. Local HTTP controls cover 8/32 independent
datasets, in-flight fragment rotation, and zero extra storage requests while a
lease remains fresh. This is not a claim of in-flight renewal for every engine
or already-open Parquet/stream handles, nor live AWS/Alpha qualification.
An already-open Parquet handle retains its original credentials. Real HTTP tests
over 8/32 row groups deliberately expire those credentials after the footer GET:
the scan fails, and reopening the retained file with a fresh read owner recovers
the exact rows/schema using the original binding revision. A changed current
revision is not substituted. This is read-attempt recovery, not transparent
mid-handle renewal or proof of whole-job recovery for every consumer.

## Recovery-frontier indexes

`AuthoritativeFrontierIndexWriter` consumes exact, authenticated state-census
Arrow batches and produces one immutable identity index. The index is bound to
the census checkpoint URI, VersionId, payload digest/size/content type, manifest
digest, catalog root, protocol, and hold fence. It rejects marker filenames,
malformed or stale state identities, duplicate/out-of-order census rows, and
mismatched cardinality.

`ExactVersionFrontierIndexScan` implements the `FrontierScan` contract over
that index. It requires a binding-scoped `StorageAccessRef`, reads the exact S3
VersionId once, verifies the byte count and SHA-256 before exposing rows, and
streams Arrow batches from bounded local disk. Its logical remote-call count is
one regardless of census or frontier cardinality. Callers must continue to run
`LeaseBatchPrecondition` before every remote mutation/publication batch.

## Retirement-cohort authorities

`write_authoritative_retirement_cohort` streams a sorted exact membership set
and a complete frozen transaction-success census into one immutable Arrow
file. `RetirementCohortIndex` memory-maps that file and performs exact binary
search over bounded record batches; a lookup is `O(log N)` and never scans or
materializes the full cohort as Python rows. Its monotonic-successor proof
streams both sorted sections in fixed Arrow chunks, proving member and success
set inclusion in `O(M + S)` time and bounded memory.

`RetirementStateEvidence` carries only state already authenticated by the
caller: member identity, normalized status, explicit commit scope, and at most
one direct transaction parent. Resolution is pure and depth-one. Unbound
sessions resolve to reversible abort; applied state with complete success
identity resolves commit-forward. Unknown, contradictory, self-referential,
or unresolved state fails closed.

## Relationship To Other Packages

Use `deltacat-client` when you want the public thin client.

Use `deltacat` when you want the thick/native package.

Use `deltacat-io-core` directly only if you are intentionally building against
the shared execution layer itself.
