Metadata-Version: 2.5
Name: dynamicio
Version: 12.0.0rc1
Summary: Panda's wrapper for IO operations
Project-URL: Documentation, https://dynamicio.readthedocs.io/en/latest/
Project-URL: Source, https://github.com/VorTECHsa/dynamicio
Project-URL: Issues, https://github.com/VorTECHsa/dynamicio/issues
Project-URL: Changelog, https://github.com/VorTECHsa/dynamicio/blob/master/CHANGELOG.md
Author-email: Christos Hadjinikolis <christos.hadjinikolis@gmail.com>, Radu Ghitescu <radu.ghitescu@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data,io,pandas
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Requires-Dist: awswrangler>=3.9.0
Requires-Dist: boto3>=1.20.24
Requires-Dist: confluent-kafka<3,>=2.4.0
Requires-Dist: fastparquet>=2024.5.0
Requires-Dist: numpy>=1.24.4
Requires-Dist: pandas>=2.2.2
Requires-Dist: pandera>=0.34.1
Requires-Dist: psycopg2-binary<3,>=2.9.3
Requires-Dist: pyarrow>=16.0.0
Requires-Dist: pydantic<3,>=1.9.2
Requires-Dist: python-json-logger>=2.0.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: simplejson>=3.17.2
Requires-Dist: sqlalchemy>=1.4.11
Requires-Dist: tables>=3.10.1
Provides-Extra: polars
Requires-Dist: polars>=2.1; extra == 'polars'
Description-Content-Type: text/markdown

[![Build and Publish](https://github.com/VorTECHsa/dynamicio/actions/workflows/build_and_publish.yml/badge.svg)](https://github.com/VorTECHsa/dynamicio/actions/workflows/build_and_publish.yml)
[![codecov](https://codecov.io/gh/VorTECHsa/dynamicio/graph/badge.svg?token=McZFwD2QPh)](https://codecov.io/gh/VorTECHsa/dynamicio)
[<img src="https://img.shields.io/badge/slack-@vortexa/dynamicio_public-purple.svg?logo=slack">](https://join.slack.com/share/enQtMzg2Nzk3ODY3MzEzNi0yNTU1ZmIyN2JkMGFhZjhhZWVjNzA2OWUzNWIyMjMyYmYzZmE4MzBjYWQ3YjdhNjU1MGU2NjFkNzMyZDllMzE2?raw=True)

<img src="https://raw.githubusercontent.com/VorTECHsa/dynamicio/master/docs/images/logo-transparent.png" alt="dynamicio — Wrap your I/O" width="500"> <img src="https://raw.githubusercontent.com/VorTECHsa/dynamicio/master/docs/images/wrapped-panda.png" alt="The dynamicio panda mascot" width="100">

**Keep your business logic independent of I/O.** dynamicio is the seam between your `pandas` or `polars` code and the places your
data lives: describe each dataset once in YAML, and your code just calls `read()` and `write()`.

The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick
which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword
arguments and which bucket a task reads this week.

**Documentation:** [dynamicio.readthedocs.io](https://dynamicio.readthedocs.io/en/latest/): guides, examples and API reference
· **Changelog:** [CHANGELOG.md](https://github.com/VorTECHsa/dynamicio/blob/master/CHANGELOG.md)

## Why dynamicio

- **Logic that does not know about storage.** Resource definitions are YAML, per environment (`LOCAL`, `DEV`, `PROD`, or your own).
  Moving a pipeline between environments never touches Python.
- **Fast, safe feedback loops.** Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and
  no mocks of external services. Compare the outputs with expected files using the built-in test harness (`dynamicio.testharness`).
- **Made for the LLM era.** A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the
  change was right. A `LOCAL` environment and expected outputs give it both, and production credentials never enter the loop.
- **Contracts for your dataframes.** A [Pandera](https://pandera.readthedocs.io) model declares columns, dtypes and validations. Bad data
  fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
- **Library behavior, not a new dialect.** Options in the YAML go to `pandas` / `awswrangler` on the pandas engine, and to the polars readers and writers
  on the polars engine, so what you know still applies.
- **Sync and async.** Read and write from plain code or from `asyncio` services.

## Supported sources

| Source | Read | Write | Formats |
|---|---|---|---|
| Local files | yes | yes | `parquet`, `hdf`, `json`, `csv` |
| S3 (single object or prefix) | yes | yes | `parquet`, `hdf`, `json`, `csv` |
| PostgreSQL | yes | yes | tables |
| AWS Athena | yes | no | SQL query results |
| Kafka | no | yes | messages |

The pandas engine is the default. The polars engine runs the same IO classes on polars frames. It reads and writes parquet, CSV and JSON
files on local and S3 resources, and it reads and writes Postgres. It reads Athena and writes Kafka. One process uses one engine. See
[the polars engine](https://dynamicio.readthedocs.io/en/latest/guides/polars-engine/).

## Install

```shell
pip install dynamicio
# or, in a uv project
uv add dynamicio
# with the polars engine
pip install "dynamicio[polars]"
```

Python 3.11 to 3.14.

## See it work

Describe the resource in `orders.yaml`:

```yaml
ORDERS:
  LOCAL:
    type: local
    local:
      file_path: "[[ DATA_DIR ]]/orders.parquet"
      file_type: parquet
```

Bind it to a schema, then read and write:

```python
import pandas as pd
from pandera.typing import Series

from dynamicio import UnifiedIO
from dynamicio.config import IOConfig
from dynamicio.schema import DataModel, Field


class Settings:
    DATA_DIR = "."


class Orders(DataModel, dataset="orders"):
    order_id: Series[int] = Field(nullable=False, unique=True)
    amount: Series[float] = Field(nullable=False, ge=0)


class OrdersIO(UnifiedIO):
    schema = Orders


config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)

orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()
```

To run the same code against S3, add a `DEV` block to the YAML and set `env_identifier="DEV"`. The Python does not change.

## Where next

1. **The idea:** [why dynamicio](https://dynamicio.readthedocs.io/en/latest/getting-started/why-dynamicio/).
2. **New here?** The [quickstart](https://dynamicio.readthedocs.io/en/latest/getting-started/quickstart/) explains each step above,
   and its code is run by the test suite.
3. **Core ideas:** [concepts](https://dynamicio.readthedocs.io/en/latest/getting-started/concepts/): resource definitions, IO classes, schemas.
4. **Task guides:** [reading](https://dynamicio.readthedocs.io/en/latest/guides/reading-data/),
   [writing](https://dynamicio.readthedocs.io/en/latest/guides/writing-data/),
   [testing pipelines](https://dynamicio.readthedocs.io/en/latest/guides/testing-pipelines/).
5. **Reference:** [API reference](https://dynamicio.readthedocs.io/en/latest/reference/).

## Community and contributing

- Questions and bugs: [GitHub issues](https://github.com/VorTECHsa/dynamicio/issues)
- Contributions are welcome: [CONTRIBUTING.md](https://github.com/VorTECHsa/dynamicio/blob/master/CONTRIBUTING.md)
- Source: <https://github.com/VorTECHsa/dynamicio> · PyPI: <https://pypi.org/project/dynamicio/>
- License: Apache-2.0 ([LICENSE](https://github.com/VorTECHsa/dynamicio/blob/master/LICENSE))

Original logo illustrated by [Nick Loucas](https://www.linkedin.com/in/nickloucas/); the refreshed logo retains its circular I/O motif.


# Changelog

## v12.0.0

- breaking: `write()` on the pandas engine casts the frame before its validations and metrics, in the order of the polars engine. A validator and a metric thus see the columns of the schema with the dtypes of the schema. In 11.x they saw the frame of the caller, and the cast came after them
- breaking: a schema column that is not in the frame raises `ColumnsDataTypeError` on the pandas engine, as on the polars engine. 11.x raised `KeyError`. This applies to reads and writes. For example, a pandas read of a JSON file without a schema column, or without records, raises `ColumnsDataTypeError`. A CSV or parquet read still raises the error of its reader
- breaking: dynamicio needs Python 3.11 or later. polars 2.1 needs Python 3.11, and Python 3.10 reaches its end of life in October 2026. CI runs Python 3.11 to 3.14, and the `lowest` dependency-range job runs on Python 3.11
- feat: an engine is the dataframe library of a process. `DynamicDataIO` sends its read, write, cast, validations and metrics to the engine, in the same order as before. The pandas engine is the default and runs the code of 11.x, so a pandas process does not change and does not import polars. The engine classes are in `dynamicio.engines`
- feat: `dynamicio.configure(engine="pandas" | "polars")` sets the engine. The first read or write fixes it and logs it once. After that, another engine raises `EngineConfigurationError`. An engine that dynamicio does not have also raises `EngineConfigurationError`, and the polars engine without polars installed raises `ImportError`
- feat: the polars engine takes `pandera.polars` models and checks frames. It reads and writes the `local`, `s3_file`, `s3_path_prefix` and `postgres` resources, reads `local_batch` and `athena`, and writes `kafka`. A Kafka read and an Athena write raise the `NotImplementedError` of the pandas engine
- feat: the polars engine reads and writes parquet, CSV and JSON files on a `local` or an `s3_file` resource, and reads a `local_batch` resource. A file that it writes has the bytes of the pandas engine of 11.1.0 for CSV and JSON. A parquet file has the arrow types, the pandas metadata and the compression of `DataFrame.to_parquet`, so the pandas engine reads it back with the pandas dtypes of the model. A nullable integer column keeps integers above 2**53 in a CSV or JSON write, `decimal: ","` reads back the floats that it writes, and a parquet write with `index: false` has the pandas metadata of `to_parquet(index=False)`
- feat: the polars engine reads a file of the pandas engine. A CSV column that is not a number or a Boolean reads as a string, so `007` keeps its text, and the cast then parses the datetimes, the durations and the Booleans. A JSON integer under a temporal column is an epoch in `date_unit`. A JSON file without records reads as a frame without rows, with the columns of the model
- feat: `EngineResourceError` for a file type or an option that the polars engine cannot read or write, for example an `hdf` file, a `dtype` option, a JSON orient other than records, a CSV `na_values` map for each column, or a CSV read without a header line and without `names`. The polars engine raises it when the IO object is built, and again at each read and write. It is also a `ValueError`
- feat: `scan()` on the polars engine gives a `LazyFrame` of a `local`, `local_batch`, `s3_file` or `s3_path_prefix` resource, with the casts of the model in its plan. Parquet and CSV use `scan_parquet` and `scan_csv`, so a filter or a selection reads only the row groups and the columns that it needs. A JSON scan reads the file eagerly and then calls `lazy()`. A failed cast raises when the frame is collected. A scan does not validate or log metrics, and logs a warning if the IO object asks for them. A missing file or key raises the error of `read()`, and `missing_ok=True` gives `None`
- feat: the polars engine reads an `s3_path_prefix` resource with `path_prefix` or `dynamic_file_path`, for parquet, CSV and JSON. `read()` collects the scan, so both give the same frame. The parquet scan reads the footer of each object: a column has the widest type of the footers, as `pd.concat` gives it, and a type pair that does not upcast raises `ColumnsDataTypeError`. `Duration` columns give the finest unit. The objects without rows do not count when the types do not upcast, as on the pandas engine. An object that lacks a column gets nulls and the warning of the pandas engine, and a column that no object has raises the `ValueError` of the pandas engine. A prefix without objects raises `NoObjectsFoundError`
- feat: the polars engine reads S3 with the credentials and the region of the boto3 session of dynamicio, and refreshes them when they expire. A boto3 endpoint that is not an AWS endpoint, for example a local S3 stack, also goes to polars
- feat: the options `no_disk_space`, `partition_cols`, `mode` and `allow_delete` are accepted on an `s3_path_prefix` resource of the polars engine, and refused on the other resources
- feat: the CSV `names` option on the polars engine names the columns of the file, as on pandas. Without `header`, the first line is data, and `header: 0` replaces the names of the header line. A file with another number of columns raises the `SchemaError` of polars
- feat: a missing S3 key on the polars engine raises the error of the pandas engine: `S3KeyNotFoundError` for parquet, and `ObjectNotFoundError` for CSV and JSON. `missing_ok=True` gives `None`
- feat: the polars engine casts a frame to the dtypes of its model, and changes float NaN to null. A `LazyFrame` gets the casts in its plan. A write changes float NaN to null before its validations and metrics, so a NaN fails `has_no_null_values` as on the pandas engine. A column that does not cast, a schema column that is not in the frame, and a column that is not in the schema raise `ColumnsDataTypeError`. A string column parses to a `Datetime`. The cast messages go to the `dynamicio.core` logger. The engine ignores `strict_dtype_check`
- feat: the polars engine runs the 9 validators of the pandas engine, with the same names, messages and values, and the Pandera column checks (`str_matches`, a custom `@pa.check`). It also runs `Config.unique`, `Config.ordered` and `@pa.dataframe_check`, with the keys and the message template of the pandas engine. For a dataframe check, the count is the number of failing rows, or 1 for a check that returns one bool. `Config.unique` takes a list of column groups. `is_in` accepts a null when the pandas `isin` accepts it. All of them run in one `pl.collect_all`, and the failures raise one `SchemaValidationError`. `PolarsValidations.validator` registers a polars validator, as `@validator` does for the pandas engine
- feat: the polars engine logs the 8 metrics of the pandas engine, with the same `METRIC` lines. It computes all of them in one `select`, after the validations. For an empty or all-null column, `Min`, `Max`, `Mean`, `Std` and `Variance` give NaN, and `Counts` and `UniqueCounts` give the counts of the pandas engine, such as 0 for an empty column. `CountsPerLabel` logs 0 for an `Enum` category that is not in the column. `PolarsMetrics.metric("Name")` registers a polars metric, as a `Metric` subclass does for the pandas engine
- feat: `dynamicio.schema.polars_dtypes.DtypeMap` is the map between the pandas and the polars dtypes. It has the signed and unsigned integers, the floats, `Boolean`, `String`, `Categorical`, `Enum`, `Datetime` (`ms`, `us`, `ns`, with a time zone) and `Duration`. `datetime64[s]`, `Date`, `List`, `Decimal` and the other dtypes have no pair, and a model with one raises `ValueError` at compile
- feat: `Field(pandas_dtype="Int64")` sets the pandas dtype of a column of a polars model, for example a nullable integer or `string[pyarrow]`. It must be a pair of the polars dtype of the column. A column without it takes the pandas dtype of the reverse map, for example `int64` for `Int64` and `object` for `String`. A pandas model with `pandas_dtype` raises `ValueError`
- feat: `dynamicio.schema.polars.DataModel` is the base class of a polars model, with the same `dataset` and `table` class keywords as `dynamicio.schema.DataModel`. `ModelCompiler` compiles a `pandera.polars` model. The `data_type` of a compiled column is a pandas dtype name, from the dtype map
- feat: `DynamicDataIO.scan()` and `sink()` read and write a lazy frame. On the pandas engine they raise `NotImplementedError`, with a message that names the polars engine. The refusal does not fix the engine
- feat: `ModelEngineMismatchError` for a model of the other engine, at the creation of an IO object and at each read and write. `WrongFrameError` for a frame of the other engine at `write()` and `sink()`, before any I/O. Both are also a `TypeError`, and `ModelEngineMismatchError` is also a `ValueError`, as the 11.x error of a polars model. A refused read or write does not fix the engine
- feat: `sink()` on the polars engine writes a `LazyFrame` to a `local`, an `s3_file` or an `s3_path_prefix` resource. It runs the write, the validations and the metrics in one `pl.collect_all`. A parquet sink streams the rows to a temporary file. It then copies the row groups to a file with the arrow types, the pandas metadata and the options of `write()`. That file thus has the footer and the row groups of `write()`. A CSV, a JSON or a prefix sink collects the frame and calls the writer of `write()`, so the bytes are equal
- feat: a failed validation of `sink()` leaves the old file or object. A local parquet sink over a file keeps the mode of the old file, as `write()` does. `sink()` logs the metrics only after the validations pass. A failed cast in the plan raises `ColumnsDataTypeError`
- feat: the polars engine writes an `s3_path_prefix` resource with the prefix writer of the pandas engine. Thus `partition_cols`, `mode` and `allow_delete` give the same objects and delete the same old objects. A prefix read on either engine does not read the `col=value` directories back as a column
- feat: `dynamicio.engines.s3_upload.S3Upload` uploads a parquet file of the polars engine to S3 with the `bucket-owner-full-control` ACL. An object above 64 MiB goes up in parts of 64 MiB, 4 at a time. A failed part aborts the upload, so S3 keeps no part and the old object does not change. A broken connection starts the upload again from the start
- feat: `DynamicDataIO.footer()` reads the arrow schema, the metadata and the row count of a `local` or `s3_file` parquet file, without its rows. It is the same on the two engines and does not fix the engine. On S3 it reads the end of the object with ranged `GetObject` calls. `missing_ok=True` gives `None` for a missing file
- feat: `sink()` on the polars engine takes a list or a tuple of lazy frames, the parts of one frame in their order. Each part gets the schema columns and the casts of the model, so the parts can have other types of a column and another column order. An empty list raises `ValueError`
- feat: the polars engine reads a `postgres` resource with the query of the pandas engine, so `sql_query` and `params` (in the style of the driver) give the same rows. A `pd.read_sql` option that changes the frame, for example `dtype`, raises `EngineResourceError`
- feat: the polars engine writes a `postgres` resource with the COPY of the pandas engine, in chunks of 100,000 rows and in one transaction. Each chunk changes to pandas with the pandas dtypes of the model, so the NULL rule and the text of each value do not change. A failed chunk rolls back the write
- feat: the pandas engine maps `datetime64[us]` and `datetime64[ms]` to a Postgres `timestamp`, as `datetime64[ns]`
- feat: the Kafka option `key_column` names the column of the message keys on the two engines. The column is not in the message values. On the pandas engine it gives the bytes of `set_index(key_column)`. A value that is not a column name, or a column that the frame does not have, raises `ValueError` before the first message
- feat: the polars engine writes a `kafka` resource with the producer of the pandas engine. The resource needs `key_column`, and a missing or wrong `key_column` raises `EngineResourceError` when the IO object is built. The frame goes to pandas in chunks of 100,000 rows, with the dtypes of the model. The user callables thus get the values and types of the pandas engine. The producer flushes once
- feat: the polars engine reads an `athena` resource with the awswrangler query of the pandas engine, and casts the result to the model
- feat: a fork of a process that imported polars refuses the polars engine, because polars can deadlock in a fork. Its first polars read or write and `configure(engine="polars")` raise `EngineConfigurationError`, with a message that names `spawn` and `forkserver`. The fork gets a new engine lock and can still use the pandas engine
- feat: dynamicio does not set the number of polars threads, and `import dynamicio` does not import polars. `POLARS_MAX_THREADS` set after `import dynamicio` thus applies
- feat: the `dynamicio[polars]` extra installs `polars>=2.1`
- feat: `dynamicio.testharness` reads and compares the outputs with the engine of the process. On the polars engine it reads parquet, CSV and JSON records with the polars readers, and compares with `polars.testing.assert_frame_equal` under the same `Expectations`. A custom assertion gets polars frames. Without `sort_by`, `check_row_order=False` compares the rows of a polars frame as a multiset. `list_columns_to_sort` sorts the lists of a `List` or `Array` column and keeps the values of another column, as on pandas. A polars error of a comparison, for example a sort of an `Object` column, fails that comparison, and the harness checks the other outputs
- feat: the pytest plugin adds the option `--dynamicio-engine=pandas|polars`, which calls `dynamicio.configure()` before the first test
- feat: `dynamicio migrate --engine polars` writes a polars model for each pandas model of a module or a package, checks the parity of each model, and writes a report. The report lists the checks to port, a model with an index, the resource definitions that the polars engine refuses, and the CSV definitions without an `index` option. The command does not change the resource definitions
- test: the demo pipeline on the polars engine, in `demo/polars_pipeline`. It reads the inputs of the pandas demo, and its outputs are equal to the expected outputs of the pandas demo. A CI job runs it with `--dynamicio-engine=polars`.
- feat: `dynamicio.configure(engine="polars")` and `dynamicio migrate --engine polars` raise `ImportError` if polars is older than 2.1, the minimum of the `polars` extra. The check reads the version of the installed package and does not import polars. The message names `pip install "dynamicio[polars]"`.
- test: an install without polars. A test in a new process blocks the polars import, and checks that the modules of dynamicio, the pandas engine and the CLI work. The dependency-ranges CI jobs install dynamicio without the `polars` extra, and a step checks that polars is not installed.
- fix: on the polars engine, an integer, unsigned integer or Boolean column with a null writes the nullable pandas dtype (`Int64`, `UInt8`, `boolean`) to the pandas metadata of a parquet file, also when the model declares a numpy dtype. The pandas engine of 11.x then reads the exact values. Before, pandas read `float64` or `object`, and an integer above 2**53 lost digits. A CSV or JSON write of such a column no longer raises `cannot convert NA to integer`.
- fix: the polars engine gives polars the S3 endpoint of the boto3 client for each endpoint other than the public endpoint of a region. A VPC interface endpoint (`*.vpce.amazonaws.com`), a FIPS endpoint and a dual-stack endpoint thus reach polars. Before, polars used the public endpoint for each `amazonaws.com` host.
- fix: `footer()` on S3 sends the `ETag` of its first ranged `GetObject` in `IfMatch` on each later call. If another writer replaces the object between two calls, the read raises the `PreconditionFailed` `ClientError` of S3. The footer thus never mixes the bytes of two versions of the object
- docs: the polars guide has runnable examples of a configuration, a model, a read, a write, a scan and a sink. It also has examples of a footer, a cast error, a validation error, a validator and a metric. The docstrings of `configure`, `Field`, the polars `DataModel`, `scan`, `sink` and `footer` have `Examples` sections. The test suite runs these examples. The S3, Postgres, Kafka and Athena examples show the YAML and the call, and do not run. The README, the home page and the installation, why-dynamicio, sources and testing pages describe the polars engine. The API reference adds `Field`, the pandas and polars `DataModel` classes and the CLI
- docs: the guides have a section for an upgrade from version 11 to version 12. The examples use `s3_file` in place of `s3`. The error table lists `EngineResourceError`, `DatasetNameNotFoundError` and `TableNameNotFoundError`. The polars guide lists the options, scans and sinks that it refuses. The reference adds the pandas engine, the schema modules and the pytest plugin
- chore: the floci integration tests upload their coverage to Codecov with the `integration` flag.

## v11.1.0

- fix: the checks for an IO class without `schema`, an IO class without a mixin for its resource type, and a URL that is not `s3://` raise `SchemaNotDeclaredError`, `MissingMixinError` and `InvalidS3UrlError`. They used `assert`, which `python -O` removes. Each error is also an `AssertionError`, so code that catches `AssertionError` does not change
- fix: `session_for` disposes its engine after use, also when the close of the session raises, so the connection pool of each Postgres read or write closes its connections at once
- fix: `pickle_protocol` holds a lock while it changes the global `pickle.HIGHEST_PROTOCOL`. Two hdf writes on different threads could restore the value of each other, so that the protocol of one stayed in place for the rest of the process
- fix: a missing S3 key of a parquet or hdf `file_path` raises `S3KeyNotFoundError`, so `missing_ok=True` returns `None` for it as it does for csv and json. For hdf, and for a parquet read that awswrangler does, S3 answers the same bare 404 for a missing bucket, so the read checks the bucket and raises its `ClientError` for a missing bucket. The error survives `pickle` and `copy`, so a read in a process pool worker gives it back to the caller. The error is a `FileNotFoundError` and also the botocore `ClientError` that S3 returned, with its `response`, so code that catches `ClientError` does not change. Code that catches the narrower `NoSuchKey` class of a boto3 client must catch `ClientError` or `S3KeyNotFoundError`. A parquet read with `filters` still raises the pyarrow `FileNotFoundError`. A parquet read with an option that only awswrangler takes, such as `s3_additional_kwargs`, also raises `S3KeyNotFoundError` for a missing key
- feat: a parquet `s3_path_prefix` write takes the `mode` option of `awswrangler.s3.to_parquet`: `append` (the default), `overwrite` or `overwrite_partitions`. With pyarrow, each write gives its files new names, so a second write of the same rows with `append` adds them again. fastparquet names the files of each partition `part.0.parquet` and onward, so a write replaces the files of the same name, as in 11.0.0. `overwrite` deletes the objects that were under the prefix, `overwrite_partitions` only those in the partitions that the write writes to. These 2 modes delete objects only when the resource also sets `allow_delete: true`. An `allow_delete` that is not a bool, such as the YAML string `"false"`, raises `TypeError`. Up to 11.0.0, a prefix write dropped `mode` with a warning and appended, so a resource that already sets `mode` and not `allow_delete` still appends, with a warning, also for a value that is not one of the 3. With `allow_delete: true`, a value that is not one of the 3 raises `ValueError`. A delete mode raises `ValueError` when the key prefix is empty, because the prefix is then the whole bucket. `overwrite` of a DataFrame with no rows raises `ValueError` and deletes nothing: pyarrow writes no file for it, so the write would delete every object under the prefix. The old objects are deleted after all new objects are uploaded. `overwrite_partitions` finds the written partitions from the `col=value` directories of `partition_cols`, also when `partition_cols` is one column name and not a list, so a file directly under the prefix does not make it delete the other partitions. `overwrite_partitions` with an empty `partition_cols` raises `ValueError`, because every key would then be in one partition. `overwrite_partitions` with the fastparquet engine, also through `engine: auto` and the pandas option `io.parquet.engine`, raises `ValueError`: fastparquet does not encode the partition values in the keys, so the value `a/b` writes to the directory of the value `a`, and dynamicio would delete the old objects of `a`. The listing of the old keys repeats after a broken connection, as the uploads and deletes do. `s3_sync_up` returns the keys of the objects it uploads, where it returned `None`. dynamicio lists the old keys before the upload: if another writer replaces one of them during the write, dynamicio deletes that object, so let only one writer write to a prefix at a time
- fix: a csv, json or hdf `path_prefix` or `dynamic_file_path` reads the objects of its sub-prefixes, as a parquet one does. The download keeps the key layout, so a sub-prefix was a directory and the read raised `IsADirectoryError`: a `dynamic_file_path` with `**/` always failed for these types. The objects are now read in key order, the order of the parquet read. Before, the order of the rows was the order of the files in the temporary directory, which the file system chose
- fix: a `local_batch` source with no matching file raises `NoObjectsFoundError`, as an S3 prefix does, so `missing_ok=True` returns `None` for it. The error is a `FileNotFoundError` and also a `ValueError`, so code that catches the pandas `ValueError: No objects to concatenate` does not change. A relative `path_prefix` or `dynamic_file_path` reads its files: the read joined the prefix to the paths a second time and raised `FileNotFoundError`
- fix: a Kafka write polls the producer after each message, so a failed delivery raises during the write. When the local queue of the producer is full, the write waits for deliveries and sends the message again, where it raised `BufferError`. The new `flush_timeout` option is the most seconds that a write waits for Kafka with no progress, and a smaller local queue is progress. An error event of the producer, such as a refused connection, is not progress. A full queue that has no room for a message in that time, or a final flush that delivers no message in that time, raises `TimeoutError`. Without it a write waits until every message is delivered, as before. A `flush_timeout` that is not a number of 0 or more, such as `-1`, NaN or the YAML string `"30"`, raises `ValueError` before the first message. A falsy callable in one of these 4 options is used, as in 11.0.0, and not replaced by the default. The write no longer removes `key_generator`, `document_transformer`, `key_serializer` and `value_serializer` from the options of the IO object. A message that does not fit in the empty queue raises `BufferError` with the name of the setting to change, and does not retry without end
- fix: a Postgres write keeps a carriage return (`\r`) in a text value. The temporary file of the `COPY` read it back as a line feed (`\n`), so the table got a different value. The file is now UTF-8 for every locale
- fix: `is_in(..., exact=True)` accepts a list of values. It compared a set with the list and raised `TypeError`. A model compiles `Field(exactly=...)` to a set, so models were not affected
- perf: `write()` no longer copies the data of a frame that has only the schema columns. A check compared a set with a tuple, was always true, and selected the columns on each write. The output does not change, and the caller's frame keeps its dtypes

## v11.0.0

Pandera `DataFrameModel` classes are now the only way to declare a schema. Upgrading from 8, 9 or 10: see [Upgrading](https://dynamicio.readthedocs.io/en/latest/guides/schemas-and-validation/#upgrading); `dynamicio migrate` converts the YAML schemas of version 8 and earlier.

- breaking: dict schemas (`schema = {...}`, `schema_validations`, `schema_metrics`) and YAML schemas (`schema = SCHEMA_FROM_FILE`, and the `schema:` block of a resource definition) are removed. An IO class whose `schema` is anything but a `DataFrameModel` or `SCHEMA_FROM_ARGUMENT` raises a `TypeError` when the class is defined, and so does a class that still sets `schema_validations` or `schema_metrics` next to a model (the model's own would replace them). A resource definition with a `schema:` key raises a `ValueError` when it loads. `SCHEMA_FROM_FILE` is removed from `dynamicio.core` and `SafeDynamicSchemaLoader` from `dynamicio.config.io_config` (importing either raises an `ImportError` that names the replacement), and `IOBinding.dynamicio_schema` / `IOEnvironment.dynamicio_schema` with it. `SCHEMA_FROM_ARGUMENT` and the `schema=` argument are unchanged
- feat: `DataModel` (`from dynamicio.schema import DataModel, Field, Metric`) is the base class of a model: a Pandera `DataFrameModel` that names its dataset and table in class keywords, `class Orders(DataModel, dataset="orders", table="orders_v2")`, so the body holds only columns and the editor completes the keywords. Any other class keyword raises a `TypeError` that names it. A plain `pa.DataFrameModel` declares the same with the dunders `__dataset_name__` and `__table_name__`. Validation, metrics and casting checks stay arguments of the IO call (`apply_schema_validations`, `log_schema_metrics`, ...), unchanged from 10.x, so a call site always shows what it does
- feat: `Field` declares a column: `pa.Field` with dynamicio's validations and metrics as named, documented arguments (`nullable=` required, `unique`, `gt`/`ge`/`lt`/`le`, `isin`, `exactly`, `max_null_fraction`, `metrics=[Metric.MIN, ...]`); any other `pa.Field` argument passes through, `metadata=` is refused. A range is its two bounds (`ge=1941, le=NEXT_YEAR`): bounds are not compared on definition, so they may come from settings, where Pandera's `in_range` would raise. `Field` returns a `pa.Field`, so it also declares the columns of a plain `pa.DataFrameModel`; it is the one public way to declare dynamicio's metrics and validations, and `from dynamicio.schema import metrics` is the metrics module again (as in 10.x), not a helper
- feat: `Metric` is a plain `Enum`, no longer a `str` subclass, so an editor offers only its eight members after `Metric.`; its `.value` is still the name the `METRIC` log line reports
- breaking: every model must name its dataset (`dataset=` or `__dataset_name__`), and nothing is inferred: a model without one raises `DatasetNameNotFoundError` when the IO object is created, where 10.x labelled it with its class name. The names are not inherited. `Config.name` is no longer read, and a model that relied on the class-name label must now write it, upper-cased in logs and `METRIC` lines as before
- breaking: the Postgres table name is `table=` (or `__table_name__`), the exact name of the table in the database, and no longer the dataset name. A model used with a Postgres resource without it raises `TableNameNotFoundError` when the IO object is created (a read with `sql_query`, which names its own tables, needs none), and a read of a table that does not exist logs which one to correct. A dotted name is a table of a Postgres schema (`table="sales.orders"`), and every statement quotes the name as Postgres needs it, so a mixed-case `table="Orders"` is truncated, deleted from and copied into as `"Orders"`; writes load the rows with `COPY ... FROM STDIN` instead of `copy_from`, which quoted a schema-qualified name as one identifier
- breaking: the raw `metadata={"dynamicio": {"metrics": [...], "validations": {...}}}` dict is rejected with a `ValueError` that names `Field`
- breaking: the `match_all` option of the `is_in` validator, which read the wrong way round (`match_all=False` was the check that every value is present), is now `exact` with the opposite meaning: `is_in(..., exact=True)` requires exactly the allowed values, the default `exact=False` only a subset of them. `Field(exactly=[...])` compiles to `exact=True` and `isin=[...]` to the default; `dynamicio migrate` translates a YAML `match_all`. Code that calls `dynamicio.schema.validations.is_in` directly with `match_all` must switch
- feat: `dynamicio migrate` is the path from version 8: it converts YAML schemas to `DataModel` classes declared with `Field`, `dataset=`, and `table=` for every schema that a resource definition under `--scan` binds to a Postgres environment (the old schema name was the table name), checks each converted model compiles back to an equivalent schema, and writes `MIGRATION_PROMPT.md` listing the remaining call sites. Dtypes are declared as types (`np.uint32`, `pd.UInt32Dtype`), not calls. A YAML `is_between` becomes its two bounds, merged with the other bounds of the column into the tightest one per side: such a column now reports `is_greater_than`/`is_lower_than` (or the `_or_equal` forms) instead of one `is_between` in `SchemaValidationError`. A `[[ VARIABLE ]]` null threshold migrates like any other bound (`Field(max_null_fraction=...)` checks a number on definition and keeps a placeholder as it is). Class and module names are valid Python whatever the file is called (`20261113.yaml` becomes `Dataset20261113`), and a column named like a name the module uses (`Field`, `Metric`, `Series`, a dynamic variable) gets a `column_` attribute with an `alias`. The prompt lists the `name =` lines of `class Config` blocks (which 11 no longer reads) instead of every `class Config`, and the search skips hidden, `node_modules`, `__pycache__`, `venv` and `site-packages` directories
- feat: the `dynamicio` command line (`dynamicio -s|-b -p PATH -o DIR`) writes a Pandera model module per dataset (`<dataset>.py`, prefixed `dataset_` where the name would not be importable) instead of a YAML schema
- docs: the schemas guide, quickstart, examples and README describe models only; new ADR 0009. The demo project's models are `DataModel` classes declared with `Field`; `Bar`, bound to Postgres in DEV and PROD, names its table, which a test now checks for every environment

## v10.4.1

- docs: the `Documentation` project link of the package, and the README, point to the documentation on Read the Docs (https://dynamicio.readthedocs.io/en/latest/), and the documentation home page shows the panda mascot next to the logo
- docs: remove the changelog sections of the `v8.1.0` release candidates.

## v10.4.0

- feat: the constructor of an IO class takes `missing_ok: bool = False`. With `missing_ok=True`, `read()` and `async_read()` return `None` and log a warning when the reader raises a `FileNotFoundError`. The caller handles the `None`, so the setting is in the code of the caller
- chore: remove `missing_ok` from the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with a `missing_ok` key now fails to load, because dynamicio reads the key as an environment name. `IOBinding.missing_ok` and `IOEnvironment.missing_ok` are removed, and the `dict()` output of a binding has no `missing_ok` entry. Move the setting to the constructor of the IO class
- docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (`docs/`), built with MkDocs for Read the Docs (`.readthedocs.yaml`; versioned `latest`, `stable` and per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported.
- docs: docstrings that the API reference could not parse (`Returns:`, `Args:` and `Raises:` entries of `DynamicDataIO.async_read`, `IOConfig`, `WithS3`, `WithPostgres`, `WithKafka`, `allow_options`, `args_of` and `OutputSandbox`) are corrected. `args_of` gains a `Callable` annotation on `*funcs` and `OutputSandbox` annotates `directory`; behaviour is unchanged.
- chore: the package metadata gains `Documentation`, `Source` and `Issues` project URLs next to `Changelog`. The `Documentation` URL points at the `docs/` directory until the Read the Docs site is confirmed.
- ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (`twine check`, tag against version) and publishes exactly those files. A release tag must be `vX.Y.Z` or `vX.Y.Z-rc.N`, a stable tag must be on `master` and have a changelog section, and the GitHub release notes are the changelog section plus links (`.github/tools/release_tools.py`, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated.
- chore: the `docs` dependency group no longer includes `mkdocs-techdocs-core` and stays on `mkdocs<2`; `docs/requirements.txt` pins the documentation tooling from `uv.lock`.

## v10.3.0

- chore: remove the `magic-logger` dependency. Each dynamicio module logs through `logging.getLogger(__name__)`, with the same logger names as before. `magic-logger` has only a source distribution, so an install with `uv sync --no-build` stopped with an error. `magic-logger` also called `inspect.stack()` on each log call. `dynamicio.logger` is a `logging.Logger` for the logger `dynamicio`, and it does not have the `dict_config`, `file_config`, `listen` and `stop_listening` methods. The logging config of `dynamicio/__init__.py` names the logger `dynamicio`, so `disable_existing_loggers` does not disable the module loggers. If you set `DYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1` and call `logging.config.dictConfig` after the dynamicio import, name the logger `dynamicio` in your config or set `disable_existing_loggers` to `false`. If you do not, your config disables the dynamicio loggers

## v10.2.0

- feat: a resource definition can set `missing_ok: true` on a source, beside its environments. When the reader of that source raises a `FileNotFoundError` (a missing local file, `ObjectNotFoundError`, `NoObjectsFoundError`), `read()` and `async_read()` return `None` and log a warning. All other errors raise as before. The default is `false`. `IOBinding.missing_ok` and `IOEnvironment.missing_ok` expose the flag, which adds a `missing_ok: False` entry to the `dict()` output of a binding. See "Optional sources: `missing_ok`" in the README
- fix: an S3 read of a csv or json `file_path` raises `dynamicio.errors.ObjectNotFoundError("No object found at s3://...")` when the key does not exist. Before, it raised the awswrangler `NoFilesFound`, which is not a `FileNotFoundError`. `ObjectNotFoundError` is a `FileNotFoundError` and a `NoFilesFound`, so a caller can treat the key as a missing source, and code that catches `NoFilesFound` works as before. The errors of the parquet and hdf reads for a missing key do not change

## v10.1.0

- docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
- feat: resource definitions accept an optional `expected: {path, file_type, options}` key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (`IOEnvironment.expected` is `None` when absent, which adds an `expected: None` entry to the model's `dict()` output.)
- feat: the `dynamicio.testharness` package, to test a task end to end: run it in a `LOCAL` environment, then lazily compare each generated output with its `expected` file. `ExpectedOutputs` collects all mismatches and raises one `OutputMismatchError`; generated files of matching outputs are removed (the others are kept); JSON is read with the resource's `orient` (or `expected.options`); paths can carry `{placeholders}` filled from `params`; `DYNAMICIO_UPDATE_EXPECTED=1` regenerates the expected files. `Expectations` curates how strict a comparison is (row/column order, dtypes, tolerances, `sort_by`, `only_columns`, `ignore_columns`, `list_columns_to_sort`, custom `assertions`; `strict()` / `lenient()` presets, `with_()` to derive variants), per session or per check; dict columns are compared by content. `FrameComparison` and `OutputFiles` are usable on their own. See "Testing Locally" in the README.
- feat: `dynamicio.testharness` additions: `ExpectedOutputs.check_each(config, key, grid)` checks a resource whose path has several `{placeholders}` for every combination of their values (`clear_each` removes their leftovers); `Expectations(check_parquet_schema=True)` also compares the arrow schema and stored pandas index of parquet outputs; `OutputSandbox(directory, *configs)` redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel; `ExpectedOutputs(environment=...)` makes every `IOConfig` that defines that environment select it inside the `with` block (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patching `env_identifier`; `check(..., environment=...)` overrides it for one check, `configs=[...]` narrows the switch, `EnvironmentSwitch(environment, *configs)` does the same explicitly and rejects an environment a config does not define. Without `environment=`, the default is the first environment the resource definitions list, whatever the process runs in. `IOConfig.instances()` and `IOConfig.default_environment` support this. The README gains "Choosing the environment" and "Recommended testing practices".
- fix: `dynamicio.testharness` review fixes: JSON Lines outputs (`options.lines: true`) are read back with `lines` (it is passed with `orient` to the JSON reader only); a column named in `sort_by`, `only_columns` or `list_columns_to_sort` that is missing from the output or the reference is reported as a failed check, not a `KeyError` that aborts the remaining checks; update mode (`DYNAMICIO_UPDATE_EXPECTED=1`) refuses to copy an output over a reference declared in another file type, leaving both files untouched.
- feat: `ExpectedOutputs.check_all(config, skip=, rules_for=, params=)` checks every resource of a config that declares `expected` (a new output is covered without touching the test; resources with unfilled `{placeholders}` are listed in `skipped`), and `ExpectedOutputs.clear_all(config)` removes the stale generated files of those resources.
- feat: a passing test run now says what it verified. `ExpectedOutputs` prints a `SessionReport` at exit (green `✓` per matched output with its rows x columns and expected file, red `✗` per failure, `↻` updated, `-` skipped), `outputs.report` returns it, `OutputMismatchError` also lists the outputs that matched, and a pytest plugin (`pytest11` entry point `dynamicio`; `-p no:dynamicio` turns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing.
- docs: the README walkthrough follows the `demo/` project: four resource definitions (`staging_input`, `staging_output`, `transform_input`, `transform_output`) with `LOCAL`, `DEV` and `PROD` environments, `expected` paths on the outputs' `LOCAL` environment, and the `staging_*`/`transform_*` configs, instead of `sample`/`actual` and `input`/`raw`/`processed`.
- docs: the `demo/` project now has one input and one output definition per runner (`staging_input`, `staging_output`, `transform_input`, `transform_output`) with `LOCAL`, `DEV` and `PROD` environments, and its end-to-end test uses `ExpectedOutputs`. The definitions use YAML anchors and merge keys (`&`, `*`, `<<:`) so `PROD` only restates what differs from `DEV`.

## v10.0.1

- fix: an S3 path-prefix read with no objects under the prefix raises `dynamicio.errors.NoObjectsFoundError("No objects found under s3://...")`. Before, it raised the pandas error `ValueError: No objects to concatenate`, which a caller cannot tell apart from other `ValueError`s. The change applies to all the file types (parquet, csv, json, hdf) and to the hdf `no_disk_space` read. `NoObjectsFoundError` is a `FileNotFoundError` and a `ValueError`, so a caller can treat the prefix as a missing source, and code that catches `ValueError` works as before. An S3 parquet `file_path` that ends in `/` and has no objects raises the same error. It raised a plain `FileNotFoundError` before. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandas `ValueError` as before

## v10.0.0

- fix: `dynamicio migrate` writes a YAML `object` column as `Series[object]`, not `Series[str]`. `object` meant "any values", and list columns declared `str` fail plain Pandera validation (and would be recast under pandas 3). Models already generated with `Series[str]` for list columns should be changed to `Series[object]`; the generated module now says where `str` is safe
- docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the `CONTRIBUTING.md` link are fixed, and the dependency list names what the package really uses. The `demo/` example datasets are generic (`orders`).
- fix: the dtype check compares dtypes instead of their names. A column already at `string[pyarrow]` (whose name is `string`) was recast on every read and logged a casting warning; a bare `category` still accepts any categorical dtype and a bare `string` any string storage (`string[python]` or `string[pyarrow]` ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before
- feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare `KeyError`. Column checks such as `str_matches`, `str_length` and custom `@pa.check` run as Pandera checks; `@pa.dataframe_check`, `Config.unique`, `Config.ordered` and index declarations run on the whole frame. Failures are raised in `SchemaValidationError` under the check's name
- feat: Pandera options dynamicio cannot honour (`Config.strict = True`, `add_missing_columns`, `on_missing_columns`, `unique_column_names`, `drop_invalid_rows`, `parsers`, `dtype`, and per column `regex`, `required=False`, `default`, `on_missing`) and non-pandas (for example polars) models raise `ValueError` when the schema is compiled, instead of being dropped. Use `strict = "filter"`. `coerce` is honoured through the compiled dtype (a nullable int becomes `Int64`)
- fix: `Series[str]` compiles to the pandas default string dtype (`object` on pandas 2, `str` on pandas 3) instead of a hard-coded `object`. Existing pandas 2 behaviour is unchanged
- feat: models written by `dynamicio migrate` set `strict = "filter"`, so `Model.validate(df)` on its own drops columns that are not in the schema, as dynamicio does. `coerce` stays off: Pandera would turn a nullable int column into `Int64`, where dynamicio casts to `int64`. Dynamicio's own validation is unchanged
- fix: no more `UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removed` on import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keep `IOEnvironment._parent` private
- fix: `WithPostgres` reads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML's `schema` when it has one (unchanged), otherwise from the model (`Config.name` is the table name). Before, a postgres resource without a YAML schema failed with `AssertionError: The schema must be specified for SQL tables`
- feat: `dynamicio migrate -o <package dir>` writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over); `-o file.py` still writes a single module. The README recommends one models module per runner stage or definitions file, and the `demo/` project now follows it (`demo/src/resources/schemas/{input,processed}.py`)
- fix: `dynamicio migrate` handles real-world schemas: mixed-type `is_in` values (incl. `null`), `is_between` bounds that are `[[ VARIABLES ]]`, dtype aliases such as `double`/`int`, and explicit default options; a failed parity check now lists the differing columns.
- refactor: schema modules now live in a `dynamicio.schema` subpackage (`schema.validations`, `schema.metrics`, `schema.models`, `schema.migrate`). `dynamicio.validations` and `dynamicio.metrics` keep working as aliases of the same module objects, and the `dynamicio.metrics` logger name is unchanged. `dynamicio.pandera_schema` and `dynamicio.migrate` (new in 10.0.0-rc.1 only) moved to `dynamicio.schema.models` and `dynamicio.schema.migrate` without aliases.
- refactor: the helpers of `dynamicio.schema` are grouped into classes (`ModelCompiler`, and for `migrate`: `SchemaComparison`, `ModelRenderer`, `LegacyDeclarations`, `MigrationPrompt`, `Migration`), and `_is_declared_dtype` is a static method of `DynamicDataIO`. The public function names (`to_dataframe_schema`, `dataframe_validator`, `migrate`, `render_model`, `render_module`, `compile_module`, `schemas_equivalent`, ...) remain as aliases, so imports keep working.
- chore: raise the `numpy` floor to 1.24.4, the oldest release Pandera 0.34 accepts (found by the `dependency-ranges (lowest)` job).
- Docs: README review. Fixed stale schema wording, the `is_between` `include_right` default, the casting-failure exception (`ColumnsDataTypeError`), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes.
- test: `tests/test_contracts/` pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style
- feat: schemas can be declared as Pandera `DataFrameModel` classes (`class MyIO(UnifiedIO): schema = MyModel`). Dynamicio's validators and metric logging are unchanged, so error payloads and `METRIC` log lines are identical to the YAML and dict styles. `pandera` is now a required dependency. Anything Pandera cannot express natively goes in `Field(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}})`. Note Pandera's default is `nullable=False`; dynamicio's YAML allowed nulls unless `has_no_null_values` was declared
- fix: when the same validation fails on several columns, `SchemaValidationError.message` now holds every column's message (joined with `; ` under the one validation-name key). Previously only the last column's message survived
- feat: `schema=` constructor argument and `SCHEMA_FROM_ARGUMENT` placeholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart of `SCHEMA_FROM_FILE`). The argument takes precedence over the class attribute
- feat: `dynamicio migrate <schemas dir> -o schemas.py` converts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writes `MIGRATION_PROMPT.md` listing the call sites to update by hand or with an LLM. The existing `-b/-s` schema generator is unchanged
- docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the `demo/` project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together under `demo/src/resources` (with `environment.py` and `constants.py` under `demo/src/config`), with `demo/tests` mirroring `demo/src`, documented in the README as the recommended pipeline project structure

## v9.2.3

- fix: `WithPostgres` builds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise a `RecursionError` during garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe for `async_read` / `async_write`. Queries and tables are unchanged.

## v9.2.2

- fix: an IO object can be read or written repeatedly. A read or write removed values from `self.options`, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example `{runner_id}`) raised `ValueError: Expected values for all fields`. Three cases gave no error. The S3 path-prefix read removed `no_disk_space`. The Athena read removed `query`, and the Postgres read removed `sql_query`, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, and `self.options` does not change. The two private methods `WithS3._read_parquet_object` and `WithS3._concat_parquet_objects` are removed. The module-level functions with the same names are unchanged

## v9.2.1

- fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the `standard` retry mode with 10 attempts, `connect_timeout=10` and `read_timeout=60`. The in-memory and file transfers (`put_object`, `upload_fileobj`, `get_object`, `download_fileobj`) are repeated, up to 5 attempts with backoff, on `ConnectionClosedError`, `ConnectTimeoutError`, `EndpointConnectionError`, `ReadTimeoutError`, `IncompleteReadError`, `ResponseStreamingError` and the `RetriesExceededError` that the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for example `ClientError`) are raised as before, and nothing changes when no error occurs

## v9.2.0

- feat: a single-file S3 parquet read with `filters` reads only the row groups whose footer statistics can match, over ranged `GetObject` requests through `pyarrow.fs.S3FileSystem` (one for each process and bucket). The rows, index and dtypes are the same as before. Reads without `filters` or with `filters=[]`, fastparquet reads, reads with a non-pyarrow `filesystem`, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrow `FileNotFoundError` or `OSError` in place of botocore `ClientError`, and a custom S3 endpoint of boto3 (`AWS_ENDPOINT_URL`) does not apply. See ADR 0004
- feat: new postgres write option `create_table` (default `False`). When `True`, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (`truncate_and_append`, `create_table`), so it drops other options with a warning, as the other writers do. A `create_table` value that is not a bool (for example the YAML string `"false"`) raises `TypeError`
- fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with `filters` on a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188)
- chore: dev dependency `moto[server]>=5` for the tests of the ranged read

## v9.1.0

- chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (`fsspec`, `libcst`, `logzero`, `no_implicit_optional`); dropped exact pins (`confluent-kafka>=2.4.0,<3`, `psycopg2-binary>=2.9.3,<3`, `python-json-logger>=2.0.1`, `simplejson>=3.17.2`); lifted the `numpy<2` ceiling (numpy 2.x passes the suite) and the `python<3.13` ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1: `pandas>=2.2.2` (JSON writes with `index=False` for `records`/`values`, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2), `pyarrow>=16.0.0`, `fastparquet>=2024.5.0`, `awswrangler>=3.9.0` (older ones cannot import against numpy 2 or pyarrow 16), `PyYAML>=6.0` (5.4.1 does not build with current toolchains). New CI job `dependency-ranges` tests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it
- fix: `python-json-logger` 3.1+ import path (`pythonjsonlogger.json`) with fallback to the old module, avoiding a `DeprecationWarning` on import

## v9.0.0

**Breaking:** Athena resources must rename `s3_staging_dir` to `s3_output` (see the first Athena entry below).

- docs: add Backstage `catalog-info.yaml` to enable TechDocs
- docs: README documents Athena I/O (resource options, `ctas_approach`, dtypes, `s3_staging_dir` migration); ADR 0003
- chore: migrate project and dependency management from Poetry to uv (PEP 621 `pyproject.toml`, `uv.lock`, `hatchling` + `hatch-vcs` build backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now use `make dev-env-setup` / `uv sync`; see `CONTRIBUTING.md` and `AGENTS.md`
- feat(RND-13624)!: `WithAthena` reads through AWS Data Wrangler (`wr.athena.read_sql_query`) instead of PyAthena, and the `pyathena` dependency is dropped. **Breaking (resource definitions):** the `s3_staging_dir` key of an `athena:` section is removed. Loading such a definition now fails with an error saying to rename it to `s3_output` (same value); nothing else changes in the YAML. `region_name` and `query` stay optional, and `database` (default `default`, Athena's own default) and `workgroup` are new optional keys. Reads use the shared per-process boto3 session unless `region_name` is set. Extra `options` are filtered to what `read_sql_query` accepts (e.g. `ctas_approach`, `unload_approach`, `athena_cache_settings`); `chunksize` is rejected with an error, because a chunked read returns an iterator rather than a DataFrame
- `ctas_approach` defaults to `False` (as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Pass `ctas_approach: true` in `options` to opt in for large results
- note: results now come back with pandas nullable dtypes (`Int64`, `string`, `boolean`) and `date` as Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source

## v8.1.1

- fix: S3 JSON reads and writes use pandas' defaults again: `lines=False` (a JSON array for `orient="records"`, as written by 7.x and earlier) unless the caller passes `lines`. 8.1.0 defaulted to JSON Lines for `orient="records"`, which could not read existing files and wrote files that 7.x readers cannot parse. Set `lines: true` in the resource `options` to opt in to JSON Lines
- fix: S3 JSON writes default `index` per orient as pandas does (`True` for `index`/`columns`/`split`/`table`, `False` for `records`/`values`) instead of always `False`, which made `orient="index"` and `orient="columns"` writes fail or drop the index
