Metadata-Version: 2.5
Name: dynamicio
Version: 10.4.0
Summary: Panda's wrapper for IO operations
Project-URL: Documentation, https://github.com/VorTECHsa/dynamicio/tree/master/docs
Project-URL: Source, https://github.com/VorTECHsa/dynamicio
Project-URL: Issues, https://github.com/VorTECHsa/dynamicio/issues
Project-URL: Changelog, https://github.com/VorTECHsa/dynamicio/blob/master/CHANGELOG.md
Author-email: Christos Hadjinikolis <christos.hadjinikolis@gmail.com>, Radu Ghitescu <radu.ghitescu@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data,io,pandas
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Requires-Dist: awswrangler>=3.9.0
Requires-Dist: boto3>=1.20.24
Requires-Dist: confluent-kafka<3,>=2.4.0
Requires-Dist: fastparquet>=2024.5.0
Requires-Dist: numpy>=1.24.4
Requires-Dist: pandas>=2.2.2
Requires-Dist: pandera>=0.34.1
Requires-Dist: psycopg2-binary<3,>=2.9.3
Requires-Dist: pyarrow>=16.0.0
Requires-Dist: pydantic<3,>=1.9.2
Requires-Dist: python-json-logger>=2.0.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: simplejson>=3.17.2
Requires-Dist: sqlalchemy>=1.4.11
Requires-Dist: tables>=3.10.1
Description-Content-Type: text/markdown

[![Build and Publish](https://github.com/VorTECHsa/dynamicio/actions/workflows/build_and_publish.yml/badge.svg)](https://github.com/VorTECHsa/dynamicio/actions/workflows/build_and_publish.yml)
[![codecov](https://codecov.io/gh/VorTECHsa/dynamicio/graph/badge.svg?token=McZFwD2QPh)](https://codecov.io/gh/VorTECHsa/dynamicio)
[<img src="https://img.shields.io/badge/slack-@vortexa/dynamicio_public-purple.svg?logo=slack">](https://join.slack.com/share/enQtMzg2Nzk3ODY3MzEzNi0yNTU1ZmIyN2JkMGFhZjhhZWVjNzA2OWUzNWIyMjMyYmYzZmE4MzBjYWQ3YjdhNjU1MGU2NjFkNzMyZDllMzE2?raw=True)

<img src="https://raw.githubusercontent.com/VorTECHsa/dynamicio/master/docs/images/logo-transparent.png" alt="dynamicio — Wrap your I/O" width="500"> <img src="https://raw.githubusercontent.com/VorTECHsa/dynamicio/master/docs/images/wrapped-panda.png" alt="The dynamicio panda mascot" width="100">

**Keep your business logic independent of I/O.** dynamicio is the seam between your `pandas` code and the places your
data lives: describe each dataset once in YAML, and your code just calls `read()` and `write()`.

The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick
which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword
arguments and which bucket a task reads this week.

**Documentation:** [guides, examples and API reference](https://github.com/VorTECHsa/dynamicio/tree/master/docs)
<!-- Replace the link above with the Read the Docs URL once the project is imported (docs/development/readthedocs-setup.md) -->
· **Changelog:** [CHANGELOG.md](https://github.com/VorTECHsa/dynamicio/blob/master/CHANGELOG.md)

## Why dynamicio

- **Logic that does not know about storage.** Resource definitions are YAML, per environment (`LOCAL`, `DEV`, `PROD`, or your own).
  Moving a pipeline between environments never touches Python.
- **Fast, safe feedback loops.** Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and
  no mocks of external services. Compare the outputs with expected files using the built-in test harness (`dynamicio.testharness`).
- **Made for the LLM era.** A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the
  change was right. A `LOCAL` environment and expected outputs give it both, and production credentials never enter the loop.
- **Contracts for your dataframes.** A [Pandera](https://pandera.readthedocs.io) model declares columns, dtypes and validations. Bad data
  fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
- **Pandas behaviour, not a new dialect.** Options in the YAML go to `pandas` / `awswrangler`, so what you know still applies.
- **Sync and async.** Read and write from plain code or from `asyncio` services.

## Supported sources

| Source | Read | Write | Formats |
|---|---|---|---|
| Local files | yes | yes | `parquet`, `hdf`, `json`, `csv` |
| S3 (single object or prefix) | yes | yes | `parquet`, `hdf`, `json`, `csv` |
| PostgreSQL | yes | yes | tables |
| AWS Athena | yes | no | SQL query results |
| Kafka | no | yes | messages |

Dataframe execution currently supports pandas only; Polars is not yet supported.

## Install

```shell
pip install dynamicio
# or, in a uv project
uv add dynamicio
```

Python 3.10 to 3.14.

## See it work

Describe the resource in `orders.yaml`:

```yaml
ORDERS:
  LOCAL:
    type: local
    local:
      file_path: "[[ DATA_DIR ]]/orders.parquet"
      file_type: parquet
```

Bind it to a schema, then read and write:

```python
import pandas as pd
import pandera.pandas as pa
from pandera.typing import Series

from dynamicio import UnifiedIO
from dynamicio.config import IOConfig


class Settings:
    DATA_DIR = "."


class Orders(pa.DataFrameModel):
    order_id: Series[int] = pa.Field(nullable=False, unique=True)
    amount: Series[float] = pa.Field(nullable=False, ge=0)


class OrdersIO(UnifiedIO):
    schema = Orders


config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)

orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()
```

To run the same code against S3, add a `DEV` block to the YAML and set `env_identifier="DEV"`. The Python does not change.

## Where next

1. **The idea:** [why dynamicio](https://github.com/VorTECHsa/dynamicio/blob/master/docs/getting-started/why-dynamicio.md).
2. **New here?** The [quickstart](https://github.com/VorTECHsa/dynamicio/blob/master/docs/getting-started/quickstart.md) explains each step above,
   and its code is run by the test suite.
3. **Core ideas:** [concepts](https://github.com/VorTECHsa/dynamicio/blob/master/docs/getting-started/concepts.md): resource definitions, IO classes, schemas.
4. **Task guides:** [reading](https://github.com/VorTECHsa/dynamicio/blob/master/docs/guides/reading-data.md),
   [writing](https://github.com/VorTECHsa/dynamicio/blob/master/docs/guides/writing-data.md),
   [testing pipelines](https://github.com/VorTECHsa/dynamicio/blob/master/docs/guides/testing-pipelines.md).
5. **Reference:** [API reference](https://github.com/VorTECHsa/dynamicio/blob/master/docs/reference/index.md).

## Community and contributing

- Questions and bugs: [GitHub issues](https://github.com/VorTECHsa/dynamicio/issues)
- Contributions are welcome: [CONTRIBUTING.md](https://github.com/VorTECHsa/dynamicio/blob/master/CONTRIBUTING.md)
- Source: <https://github.com/VorTECHsa/dynamicio> · PyPI: <https://pypi.org/project/dynamicio/>
- License: Apache-2.0 ([LICENSE](https://github.com/VorTECHsa/dynamicio/blob/master/LICENSE))

Original logo illustrated by [Nick Loucas](https://www.linkedin.com/in/nickloucas/); the refreshed logo retains its circular I/O motif.


# Changelog

## v10.4.0

- feat: the constructor of an IO class takes `missing_ok: bool = False`. With `missing_ok=True`, `read()` and `async_read()` return `None` and log a warning when the reader raises a `FileNotFoundError`. The caller handles the `None`, so the setting is in the code of the caller
- chore: remove `missing_ok` from the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with a `missing_ok` key now fails to load, because dynamicio reads the key as an environment name. `IOBinding.missing_ok` and `IOEnvironment.missing_ok` are removed, and the `dict()` output of a binding has no `missing_ok` entry. Move the setting to the constructor of the IO class
- docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (`docs/`), built with MkDocs for Read the Docs (`.readthedocs.yaml`; versioned `latest`, `stable` and per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported.
- docs: docstrings that the API reference could not parse (`Returns:`, `Args:` and `Raises:` entries of `DynamicDataIO.async_read`, `IOConfig`, `WithS3`, `WithPostgres`, `WithKafka`, `allow_options`, `args_of` and `OutputSandbox`) are corrected. `args_of` gains a `Callable` annotation on `*funcs` and `OutputSandbox` annotates `directory`; behaviour is unchanged.
- chore: the package metadata gains `Documentation`, `Source` and `Issues` project URLs next to `Changelog`. The `Documentation` URL points at the `docs/` directory until the Read the Docs site is confirmed.
- ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (`twine check`, tag against version) and publishes exactly those files. A release tag must be `vX.Y.Z` or `vX.Y.Z-rc.N`, a stable tag must be on `master` and have a changelog section, and the GitHub release notes are the changelog section plus links (`.github/tools/release_tools.py`, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated.
- chore: the `docs` dependency group no longer includes `mkdocs-techdocs-core` and stays on `mkdocs<2`; `docs/requirements.txt` pins the documentation tooling from `uv.lock`.

## v10.3.0

- chore: remove the `magic-logger` dependency. Each dynamicio module logs through `logging.getLogger(__name__)`, with the same logger names as before. `magic-logger` has only a source distribution, so an install with `uv sync --no-build` stopped with an error. `magic-logger` also called `inspect.stack()` on each log call. `dynamicio.logger` is a `logging.Logger` for the logger `dynamicio`, and it does not have the `dict_config`, `file_config`, `listen` and `stop_listening` methods. The logging config of `dynamicio/__init__.py` names the logger `dynamicio`, so `disable_existing_loggers` does not disable the module loggers. If you set `DYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1` and call `logging.config.dictConfig` after the dynamicio import, name the logger `dynamicio` in your config or set `disable_existing_loggers` to `false`. If you do not, your config disables the dynamicio loggers

## v10.2.0

- feat: a resource definition can set `missing_ok: true` on a source, beside its environments. When the reader of that source raises a `FileNotFoundError` (a missing local file, `ObjectNotFoundError`, `NoObjectsFoundError`), `read()` and `async_read()` return `None` and log a warning. All other errors raise as before. The default is `false`. `IOBinding.missing_ok` and `IOEnvironment.missing_ok` expose the flag, which adds a `missing_ok: False` entry to the `dict()` output of a binding. See "Optional sources: `missing_ok`" in the README
- fix: an S3 read of a csv or json `file_path` raises `dynamicio.errors.ObjectNotFoundError("No object found at s3://...")` when the key does not exist. Before, it raised the awswrangler `NoFilesFound`, which is not a `FileNotFoundError`. `ObjectNotFoundError` is a `FileNotFoundError` and a `NoFilesFound`, so a caller can treat the key as a missing source, and code that catches `NoFilesFound` works as before. The errors of the parquet and hdf reads for a missing key do not change

## v10.1.0

- docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
- feat: resource definitions accept an optional `expected: {path, file_type, options}` key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (`IOEnvironment.expected` is `None` when absent, which adds an `expected: None` entry to the model's `dict()` output.)
- feat: the `dynamicio.testharness` package, to test a task end to end: run it in a `LOCAL` environment, then lazily compare each generated output with its `expected` file. `ExpectedOutputs` collects all mismatches and raises one `OutputMismatchError`; generated files of matching outputs are removed (the others are kept); JSON is read with the resource's `orient` (or `expected.options`); paths can carry `{placeholders}` filled from `params`; `DYNAMICIO_UPDATE_EXPECTED=1` regenerates the expected files. `Expectations` curates how strict a comparison is (row/column order, dtypes, tolerances, `sort_by`, `only_columns`, `ignore_columns`, `list_columns_to_sort`, custom `assertions`; `strict()` / `lenient()` presets, `with_()` to derive variants), per session or per check; dict columns are compared by content. `FrameComparison` and `OutputFiles` are usable on their own. See "Testing Locally" in the README.
- feat: `dynamicio.testharness` additions: `ExpectedOutputs.check_each(config, key, grid)` checks a resource whose path has several `{placeholders}` for every combination of their values (`clear_each` removes their leftovers); `Expectations(check_parquet_schema=True)` also compares the arrow schema and stored pandas index of parquet outputs; `OutputSandbox(directory, *configs)` redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel; `ExpectedOutputs(environment=...)` makes every `IOConfig` that defines that environment select it inside the `with` block (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patching `env_identifier`; `check(..., environment=...)` overrides it for one check, `configs=[...]` narrows the switch, `EnvironmentSwitch(environment, *configs)` does the same explicitly and rejects an environment a config does not define. Without `environment=`, the default is the first environment the resource definitions list, whatever the process runs in. `IOConfig.instances()` and `IOConfig.default_environment` support this. The README gains "Choosing the environment" and "Recommended testing practices".
- fix: `dynamicio.testharness` review fixes: JSON Lines outputs (`options.lines: true`) are read back with `lines` (it is passed with `orient` to the JSON reader only); a column named in `sort_by`, `only_columns` or `list_columns_to_sort` that is missing from the output or the reference is reported as a failed check, not a `KeyError` that aborts the remaining checks; update mode (`DYNAMICIO_UPDATE_EXPECTED=1`) refuses to copy an output over a reference declared in another file type, leaving both files untouched.
- feat: `ExpectedOutputs.check_all(config, skip=, rules_for=, params=)` checks every resource of a config that declares `expected` (a new output is covered without touching the test; resources with unfilled `{placeholders}` are listed in `skipped`), and `ExpectedOutputs.clear_all(config)` removes the stale generated files of those resources.
- feat: a passing test run now says what it verified. `ExpectedOutputs` prints a `SessionReport` at exit (green `✓` per matched output with its rows x columns and expected file, red `✗` per failure, `↻` updated, `-` skipped), `outputs.report` returns it, `OutputMismatchError` also lists the outputs that matched, and a pytest plugin (`pytest11` entry point `dynamicio`; `-p no:dynamicio` turns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing.
- docs: the README walkthrough follows the `demo/` project: four resource definitions (`staging_input`, `staging_output`, `transform_input`, `transform_output`) with `LOCAL`, `DEV` and `PROD` environments, `expected` paths on the outputs' `LOCAL` environment, and the `staging_*`/`transform_*` configs, instead of `sample`/`actual` and `input`/`raw`/`processed`.
- docs: the `demo/` project now has one input and one output definition per runner (`staging_input`, `staging_output`, `transform_input`, `transform_output`) with `LOCAL`, `DEV` and `PROD` environments, and its end-to-end test uses `ExpectedOutputs`. The definitions use YAML anchors and merge keys (`&`, `*`, `<<:`) so `PROD` only restates what differs from `DEV`.

## v10.0.1

- fix: an S3 path-prefix read with no objects under the prefix raises `dynamicio.errors.NoObjectsFoundError("No objects found under s3://...")`. Before, it raised the pandas error `ValueError: No objects to concatenate`, which a caller cannot tell apart from other `ValueError`s. The change applies to all the file types (parquet, csv, json, hdf) and to the hdf `no_disk_space` read. `NoObjectsFoundError` is a `FileNotFoundError` and a `ValueError`, so a caller can treat the prefix as a missing source, and code that catches `ValueError` works as before. An S3 parquet `file_path` that ends in `/` and has no objects raises the same error. It raised a plain `FileNotFoundError` before. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandas `ValueError` as before

## v10.0.0

- fix: `dynamicio migrate` writes a YAML `object` column as `Series[object]`, not `Series[str]`. `object` meant "any values", and list columns declared `str` fail plain Pandera validation (and would be recast under pandas 3). Models already generated with `Series[str]` for list columns should be changed to `Series[object]`; the generated module now says where `str` is safe
- docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the `CONTRIBUTING.md` link are fixed, and the dependency list names what the package really uses. The `demo/` example datasets are generic (`orders`).
- fix: the dtype check compares dtypes instead of their names. A column already at `string[pyarrow]` (whose name is `string`) was recast on every read and logged a casting warning; a bare `category` still accepts any categorical dtype and a bare `string` any string storage (`string[python]` or `string[pyarrow]` ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before
- feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare `KeyError`. Column checks such as `str_matches`, `str_length` and custom `@pa.check` run as Pandera checks; `@pa.dataframe_check`, `Config.unique`, `Config.ordered` and index declarations run on the whole frame. Failures are raised in `SchemaValidationError` under the check's name
- feat: Pandera options dynamicio cannot honour (`Config.strict = True`, `add_missing_columns`, `on_missing_columns`, `unique_column_names`, `drop_invalid_rows`, `parsers`, `dtype`, and per column `regex`, `required=False`, `default`, `on_missing`) and non-pandas (for example polars) models raise `ValueError` when the schema is compiled, instead of being dropped. Use `strict = "filter"`. `coerce` is honoured through the compiled dtype (a nullable int becomes `Int64`)
- fix: `Series[str]` compiles to the pandas default string dtype (`object` on pandas 2, `str` on pandas 3) instead of a hard-coded `object`. Existing pandas 2 behaviour is unchanged
- feat: models written by `dynamicio migrate` set `strict = "filter"`, so `Model.validate(df)` on its own drops columns that are not in the schema, as dynamicio does. `coerce` stays off: Pandera would turn a nullable int column into `Int64`, where dynamicio casts to `int64`. Dynamicio's own validation is unchanged
- fix: no more `UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removed` on import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keep `IOEnvironment._parent` private
- fix: `WithPostgres` reads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML's `schema` when it has one (unchanged), otherwise from the model (`Config.name` is the table name). Before, a postgres resource without a YAML schema failed with `AssertionError: The schema must be specified for SQL tables`
- feat: `dynamicio migrate -o <package dir>` writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over); `-o file.py` still writes a single module. The README recommends one models module per runner stage or definitions file, and the `demo/` project now follows it (`demo/src/resources/schemas/{input,processed}.py`)
- fix: `dynamicio migrate` handles real-world schemas: mixed-type `is_in` values (incl. `null`), `is_between` bounds that are `[[ VARIABLES ]]`, dtype aliases such as `double`/`int`, and explicit default options; a failed parity check now lists the differing columns.
- refactor: schema modules now live in a `dynamicio.schema` subpackage (`schema.validations`, `schema.metrics`, `schema.models`, `schema.migrate`). `dynamicio.validations` and `dynamicio.metrics` keep working as aliases of the same module objects, and the `dynamicio.metrics` logger name is unchanged. `dynamicio.pandera_schema` and `dynamicio.migrate` (new in 10.0.0-rc.1 only) moved to `dynamicio.schema.models` and `dynamicio.schema.migrate` without aliases.
- refactor: the helpers of `dynamicio.schema` are grouped into classes (`ModelCompiler`, and for `migrate`: `SchemaComparison`, `ModelRenderer`, `LegacyDeclarations`, `MigrationPrompt`, `Migration`), and `_is_declared_dtype` is a static method of `DynamicDataIO`. The public function names (`to_dataframe_schema`, `dataframe_validator`, `migrate`, `render_model`, `render_module`, `compile_module`, `schemas_equivalent`, ...) remain as aliases, so imports keep working.
- chore: raise the `numpy` floor to 1.24.4, the oldest release Pandera 0.34 accepts (found by the `dependency-ranges (lowest)` job).
- Docs: README review. Fixed stale schema wording, the `is_between` `include_right` default, the casting-failure exception (`ColumnsDataTypeError`), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes.
- test: `tests/test_contracts/` pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style
- feat: schemas can be declared as Pandera `DataFrameModel` classes (`class MyIO(UnifiedIO): schema = MyModel`). Dynamicio's validators and metric logging are unchanged, so error payloads and `METRIC` log lines are identical to the YAML and dict styles. `pandera` is now a required dependency. Anything Pandera cannot express natively goes in `Field(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}})`. Note Pandera's default is `nullable=False`; dynamicio's YAML allowed nulls unless `has_no_null_values` was declared
- fix: when the same validation fails on several columns, `SchemaValidationError.message` now holds every column's message (joined with `; ` under the one validation-name key). Previously only the last column's message survived
- feat: `schema=` constructor argument and `SCHEMA_FROM_ARGUMENT` placeholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart of `SCHEMA_FROM_FILE`). The argument takes precedence over the class attribute
- feat: `dynamicio migrate <schemas dir> -o schemas.py` converts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writes `MIGRATION_PROMPT.md` listing the call sites to update by hand or with an LLM. The existing `-b/-s` schema generator is unchanged
- docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the `demo/` project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together under `demo/src/resources` (with `environment.py` and `constants.py` under `demo/src/config`), with `demo/tests` mirroring `demo/src`, documented in the README as the recommended pipeline project structure

## v9.2.3

- fix: `WithPostgres` builds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise a `RecursionError` during garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe for `async_read` / `async_write`. Queries and tables are unchanged.

## v9.2.2

- fix: an IO object can be read or written repeatedly. A read or write removed values from `self.options`, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example `{runner_id}`) raised `ValueError: Expected values for all fields`. Three cases gave no error. The S3 path-prefix read removed `no_disk_space`. The Athena read removed `query`, and the Postgres read removed `sql_query`, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, and `self.options` does not change. The two private methods `WithS3._read_parquet_object` and `WithS3._concat_parquet_objects` are removed. The module-level functions with the same names are unchanged

## v9.2.1

- fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the `standard` retry mode with 10 attempts, `connect_timeout=10` and `read_timeout=60`. The in-memory and file transfers (`put_object`, `upload_fileobj`, `get_object`, `download_fileobj`) are repeated, up to 5 attempts with backoff, on `ConnectionClosedError`, `ConnectTimeoutError`, `EndpointConnectionError`, `ReadTimeoutError`, `IncompleteReadError`, `ResponseStreamingError` and the `RetriesExceededError` that the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for example `ClientError`) are raised as before, and nothing changes when no error occurs

## v9.2.0

- feat: a single-file S3 parquet read with `filters` reads only the row groups whose footer statistics can match, over ranged `GetObject` requests through `pyarrow.fs.S3FileSystem` (one for each process and bucket). The rows, index and dtypes are the same as before. Reads without `filters` or with `filters=[]`, fastparquet reads, reads with a non-pyarrow `filesystem`, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrow `FileNotFoundError` or `OSError` in place of botocore `ClientError`, and a custom S3 endpoint of boto3 (`AWS_ENDPOINT_URL`) does not apply. See ADR 0004
- feat: new postgres write option `create_table` (default `False`). When `True`, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (`truncate_and_append`, `create_table`), so it drops other options with a warning, as the other writers do. A `create_table` value that is not a bool (for example the YAML string `"false"`) raises `TypeError`
- fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with `filters` on a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188)
- chore: dev dependency `moto[server]>=5` for the tests of the ranged read

## v9.1.0

- chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (`fsspec`, `libcst`, `logzero`, `no_implicit_optional`); dropped exact pins (`confluent-kafka>=2.4.0,<3`, `psycopg2-binary>=2.9.3,<3`, `python-json-logger>=2.0.1`, `simplejson>=3.17.2`); lifted the `numpy<2` ceiling (numpy 2.x passes the suite) and the `python<3.13` ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1: `pandas>=2.2.2` (JSON writes with `index=False` for `records`/`values`, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2), `pyarrow>=16.0.0`, `fastparquet>=2024.5.0`, `awswrangler>=3.9.0` (older ones cannot import against numpy 2 or pyarrow 16), `PyYAML>=6.0` (5.4.1 does not build with current toolchains). New CI job `dependency-ranges` tests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it
- fix: `python-json-logger` 3.1+ import path (`pythonjsonlogger.json`) with fallback to the old module, avoiding a `DeprecationWarning` on import

## v9.0.0

**Breaking:** Athena resources must rename `s3_staging_dir` to `s3_output` (see the first Athena entry below).

- docs: add Backstage `catalog-info.yaml` to enable TechDocs
- docs: README documents Athena I/O (resource options, `ctas_approach`, dtypes, `s3_staging_dir` migration); ADR 0003
- chore: migrate project and dependency management from Poetry to uv (PEP 621 `pyproject.toml`, `uv.lock`, `hatchling` + `hatch-vcs` build backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now use `make dev-env-setup` / `uv sync`; see `CONTRIBUTING.md` and `AGENTS.md`
- feat(RND-13624)!: `WithAthena` reads through AWS Data Wrangler (`wr.athena.read_sql_query`) instead of PyAthena, and the `pyathena` dependency is dropped. **Breaking (resource definitions):** the `s3_staging_dir` key of an `athena:` section is removed. Loading such a definition now fails with an error saying to rename it to `s3_output` (same value); nothing else changes in the YAML. `region_name` and `query` stay optional, and `database` (default `default`, Athena's own default) and `workgroup` are new optional keys. Reads use the shared per-process boto3 session unless `region_name` is set. Extra `options` are filtered to what `read_sql_query` accepts (e.g. `ctas_approach`, `unload_approach`, `athena_cache_settings`); `chunksize` is rejected with an error, because a chunked read returns an iterator rather than a DataFrame
- `ctas_approach` defaults to `False` (as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Pass `ctas_approach: true` in `options` to opt in for large results
- note: results now come back with pandas nullable dtypes (`Int64`, `string`, `boolean`) and `date` as Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source

## v8.1.1

- fix: S3 JSON reads and writes use pandas' defaults again: `lines=False` (a JSON array for `orient="records"`, as written by 7.x and earlier) unless the caller passes `lines`. 8.1.0 defaulted to JSON Lines for `orient="records"`, which could not read existing files and wrote files that 7.x readers cannot parse. Set `lines: true` in the resource `options` to opt in to JSON Lines
- fix: S3 JSON writes default `index` per orient as pandas does (`True` for `index`/`columns`/`split`/`table`, `False` for `records`/`values`) instead of always `False`, which made `orient="index"` and `orient="columns"` writes fail or drop the index

## v8.1.0-rc.11

- fix: an `s3_file` parquet source whose `file_path` ends in `/` (a directory of parquet objects, e.g. a downstream pipeline's outcome dataset) is read as a dataset again: all objects under it are concatenated in listing order, with the same missing-column tolerance as `s3_path_prefix`. Since rc.7 it failed with `NoSuchKey` because the single-file reader issued a `GetObject` on the `.../` key

## v8.1.0-rc.10

- fix: S3 CSV writes accept a caller-supplied `index` option (`TypeError: to_csv() got multiple values for keyword argument 'index'` since rc.1, hit by a downstream pipeline's `IO(..., index=False)`), and default to `index=True` like `DataFrame.to_csv` and the s3fs writer on master, instead of the hard-coded `index=False` of the wrangler writer
- fix: the awswrangler parquet fallback tolerates a caller-supplied `dataset` option instead of raising a duplicate-keyword `TypeError`

## v8.1.0-rc.9

- perf: parquet `no_disk_space` prefix reads use the same in-memory thread pool as the default path (~4x faster on a 3.7M-row prefix, 199s to 46s, lower peak memory); rows, order and dtypes unchanged
- fix: parquet prefix reads return rows in S3 listing order and null-filled object columns hold `None`, as `master`'s `no_disk_space` dataset read did (see rc.8 notes)

## v8.1.0-rc.8

- perf: parquet `WithS3PathPrefix` reads fetch and parse each object in memory on the download threads (no temp dir, parsing overlaps the network); ~15% faster on ~1000 files over WAN, same rows. Files are now concatenated in listing order instead of arbitrary `os.listdir` order
- fix: `WithS3PathPrefix` parquet reads no longer fail with `ArrowInvalid: No match for FieldRef.Name(<col>)` when some objects under the prefix predate a schema column. Such objects get nulls for that column (as master's `no_disk_space` dataset read did when the first listed file had the column), a warning lists the columns and how many objects lacked them, and a column missing from every object still raises `ValueError`
- fix: parquet `no_disk_space` prefix reads return rows in S3 listing order again, as `master` did (the branch had reversed it); null-filled object columns hold `None` like `master`'s dataset read, not `NaN`
- fix: `WithPostgres` logged `Started downloading table` twice per read (the query itself ran once); now logged once

## v8.1.0-rc.7

(rc.5 and rc.6 were tagged but never published: black/pylint/flake8 style checks failed. Contents are identical apart from formatting.)

- perf: reuse one boto3 session/client per process (fork-safe) instead of building a new one per awswrangler call
- perf: single-file S3 parquet reads/writes use one `GetObject`/`PutObject` over the shared client and parse in memory with the same pandas/pyarrow options as local I/O; awswrangler is only used when a wrangler-only option is passed (fixes the ~2.6x read slowdown seen in rc.1-rc.4)
- feat!: remove `awscli` and `s3fs` dependencies; `WithS3PathPrefix` now syncs with a boto3 thread pool (one request per file, concurrency 32) instead of the in-process `aws s3 sync`. No `aws` console script is installed into consumer venvs any more
- fix: S3 HDF writer honours `pickle_protocol`
- fix: S3 JSON reader supports `single_record`

## v8.1.0-rc.4

- fix: stop forcing `convert_dates=False` on local JSON reads, letting pandas' own date auto-detection run unless the caller opts out explicitly (fixes datetime dtype regression for consumers relying on auto-parsed dates, e.g. reference-data JSON)

## v8.1.0-rc.3

- fix: honor caller-supplied `orient`/`lines` for S3 and local JSON reads/writes instead of hardcoding `orient="records"` (fixes regression for consumers using `orient="index"`, e.g. reference-data JSON)
- fix: re-apply popped `orient`/`lines` options before `df.to_json` in the local JSON writer (previously silently discarded)

## v8.1.0-rc.2

- fix: stop forcing records-only orient on local JSON reads

## v8.1.0-rc.1

- feat(RND-13624): Migrate `WithS3File` to AWS Data Wrangler
- fix: address Copilot review findings on WithS3File wrangler migration
- fix: silence pre-existing pylint invalid-name warnings to unblock style-checks
- fix: satisfy black formatting and 90% coverage gate in CI
- fix: regenerate poetry.lock to match pyproject.toml
- docs: add awswrangler vs AWS CLI performance report, PR template, and skills
