Metadata-Version: 2.4
Name: yggio
Version: 0.0.2
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Typing :: Typed
Requires-Dist: pyarrow>=18
Requires-Dist: typing-extensions>=4.4 ; python_full_version < '3.11'
Requires-Dist: mypy>=1.15 ; extra == 'test'
Requires-Dist: pandas>=2 ; extra == 'test'
Requires-Dist: polars>=1 ; extra == 'test'
Requires-Dist: pytest>=8 ; extra == 'test'
Requires-Dist: tzdata ; extra == 'test'
Provides-Extra: test
License-File: LICENSE
Summary: Native Yggio schemas, dataclass field definitions, identifiers, and byte codecs
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# yggio Python bindings

```python
import os
from decimal import Decimal

from yggio import DataType, Field, MediaType, MimeType, Uri

price = DataType.decimal("18", 4)
clock = DataType.time("microseconds")
text = DataType(str)
field = Field("price", price, nullable=False)
field.set_comment("closing price")
field.set_parquet_field_id(17)
field.set_location("s3://warehouse/bars/data.arrow")
field.set_property("postgres", "type", "numeric(18,4)")
scalar = field.scalar(Decimal("12.5000"))
file = Uri.from_path(r"C:\data\prices.arrow")
encoded = Uri("https://example.test/prices.csv.gz")

assert str(price) == "decimal64(18,4)"
assert str(clock) == "time64(us)"
assert text == DataType("utf8")
assert field.into_arrow().name == "price"
assert field.parquet_field_id == 17
assert field.metadata["PARQUET:field_id"] == "17"
assert field.location.scheme == "s3"
assert field.get_property("postgres", "type") == "numeric(18,4)"
assert scalar.as_py() == Decimal("12.5000")
assert str(file) == "file:///C:/data/prices.arrow"
assert os.fspath(file) == "C:/data/prices.arrow"
assert encoded.media_type.base == MimeType("text/csv")
encoded.set_media_type(
    MediaType.from_parts(MimeType("application/json"), [MimeType("application/gzip")])
)
assert encoded.file_name == "prices.json.gz"
```

The package provides native Python views over Yggio's `DataType`, `Field`,
`MimeType`, `MediaType`, `Uri`, `Url`, and `Urn`, plus the `@scalar` decorator
for defining schemas with standard-library dataclasses. Parsing,
validation, Arrow conversion, ordering, stable hashing, and path normalization
remain owned by Rust. Python 3.10 and PyArrow 18 are the minimum supported
versions; 15 supplied the generic Arrow C-stream reader boundary, and 18 is
where the run-end-encoded map and extension-type scalar paths became correct.
`DataType.arrow_scalar(value, *, safe=True)` builds a Scalar of the projected
physical type. `Field.arrow_scalar` adds top-level nullability enforcement and returns
an already matching Scalar by identity; `safe=False` explicitly requests
PyArrow's unsafe cast rules. Conversion from arbitrary Python objects into a
PyArrow Scalar is Python-specific; native scalar validation and materialization
are shared through the default-enabled core `yggio::arrow` runtime.

The wheel set is four builds: `cp310`, version-specific, because the buffer
protocol joined the limited API at 3.11; `cp311-abi3`, one extension for every
GIL-enabled CPython from 3.11; `cp314-cp314t` for free-threaded 3.14, which has
no stable ABI; and `cp315-abi3.abi3t`, one extension for GIL-enabled and
free-threaded CPython 3.15 and later. Free threading starts at 3.14: PyO3
0.29 builds for no free-threaded CPython below it, 3.12 has no free-threaded
build, and the abi3t wheel is 3.15's, so a GIL-enabled 3.12 or 3.13 loads
`cp311-abi3`. Importing the extension on a
free-threaded interpreter leaves the GIL disabled, and every door that waits
on the core detaches from the interpreter. Immutable classes are frozen and
are read from any number of threads at once; a mutable object used by two
threads at the same instant raises `RuntimeError: Already borrowed` rather
than corrupting, so a thread takes a handle of its own.

```python
from yggio import Field, MediaType, MimeType

media = MediaType.from_parts(MimeType.CSV, [MimeType.GZIP, MimeType.ZSTD])
field = Field(
    "payload",
    "binary",
    metadata={"HTTPS:Content-Type": "text/csv; charset=utf-8"},
)
field.set_media_type(media)

assert field.mime_type == MimeType.CSV
assert field.content_encoding == "gzip, zstd"
assert field.get_property("https", "content-type") == "text/csv"
```

`MimeType` exposes the complete native known vocabulary as immutable class
constants and accepts custom validated MIME names. `MediaType()` defaults to
octet-stream, consumes iterable encodings once, and returns a detached tuple
from `encodings`; it is mutable until first hashed. Hashing locks that wrapper,
and a copy is an independent mutable value. URI, URL, and URN values use the
same rule. Field HTTP and
HTTPS input shares canonical lowercase `http:*` Arrow metadata. Raw header
properties retain parameters, while typed MIME/media, unsigned content length,
and absolute HTTP Location accessors validate without a binding-side parser.
Pair media updates are atomic and cache-aware. Hashing a `Field` likewise
locks every equality-affecting field and metadata mutation on that wrapper;
copies and pickle round-trips are independent values.

Precise Arrow layout can stay beside the logical Python annotation. Reserved
`Annotated` options work as `(key, value)` pairs or entries in one mapping:

```python
from decimal import Decimal
from typing import Annotated

import pyarrow as pa
from yggio import Field

price = Field.from_pyhint(
    "price",
    Annotated[
        Decimal,
        ("arrow_type", pa.decimal128(9, 0)),
        {"nullable": False, "metadata": {"unit": "EUR"}, "id": 7},
    ],
)

assert price.into_arrow().type == pa.decimal128(9, 0)
assert price.parquet_field_id == 7
```

```python
from __future__ import annotations

from yggio import Field, Int32Field, SerieField, integer, serie, utf8

trade_id: Int32Field = integer.int32("trade_id", nullable=False)
tags: SerieField[str] = serie("tags", utf8("item"))

assert type(trade_id) is Field
assert tags.dtype.id == "serie"
assert tags.dtype.kind == "nested"
assert trade_id.show_diff(trade_id) == "✓ equal"
```

One module per type at the package root - `yggio.integer`, `yggio.string`, `yggio.temporal` and the rest - covers every native datatype variant, and every factory is re-exported from `yggio` itself.
Its aliases preserve kind/value information for static typing while factories
return the same generic native `Field`. `equals(..., with_metadata=False)` can
ignore metadata recursively; `show_diffs` returns readable UTF-8 difference
lines and `show_diff` joins them.

```python
from decimal import Decimal

from yggio import scalar
from yggio import toml

@scalar
class Order:
    order_id: int
    price: Decimal

order = Order(42, Decimal("12.50"))
payload = toml.dumps(order)  # ordinary UTF-8 TOML bytes

assert Order.into_field().name == "Order"
assert toml.loads(payload, cls=Order) == order
```

`yggio.json`, `yggio.toml`, and `yggio.yaml` expose `dumps`, `dump` and
`loads`, and no `load`: `dumps` answers bytes, `dump` writes to an `os.PathLike`,
a path string or a text/binary file object (and answers the bytes, or UTF-8 text
with `utf8=True`, when given none), and `loads` reads bytes, string content, an
`os.PathLike` or a text/binary file object.
A source `str` is always document content; use `pathlib.Path` to name a source
location. String destinations remain paths because output has no content/path
ambiguity. JSON Lines and YAML
also expose the document streams `dumps_all`/`dump_all` and
`loads_all`/`load_all`; TOML is deliberately one document. String content uses the borrowed native text path; paths and named or
explicitly formatted I/O redirect to the native reader/writer paths without
staging a whole encoded document. For `load_all`, Python supplies only its
`read(size)`/`readline(size)` protocol: the owning Rust iterator decides JSON
Lines and YAML document boundaries, enforces cumulative limits, and reports
core byte offsets. It reads lazily under one bounded parser window, leaves the
caller-owned stream open, and is fused after its first read or parse failure.
A document carries shapes, never class names: a `set` reads back as a list and a
`uuid.UUID` as its text, and a class is constructed only through an explicit
target such as `cls=`.

```python
import pyarrow as pa

from yggio import Field, field

trade_field = Field.from_arrow_schema(
    pa.schema([
        pa.field("trade_id", pa.uint32(), nullable=False),
        pa.field("tags", pa.list_(pa.string()), nullable=False),
    ]),
    name="Trade",
)
Trade = trade_field.into_dataclass()

trade = Trade(trade_id=1, tags=["new"])

assert field(Trade) is trade_field
assert Trade.into_field() is trade_field
assert Trade.into_field().into_arrow_schema().field("trade_id").type == pa.uint32()
```

Arrow schemas import through the native `Field`, preserving exact physical
widths, metadata, dictionary state, and nested layout. `into_dataclass()`
builds a normal dataclass whose cached static `into_field()` result is that
native shape; it does not inject row conversion, codec, or Arrow methods into
the class.

Use cached `Class.into_field()` for decorated dataclasses and `field(value)`
for the general conversion funnel.

`Serie.from_arrow_array(array, field)` and `Serie.from_arrow_batch(batch, root)`
cast through the native Arrow kernel plan, `SerieReader.from_arrow_reader`
casts a stream by one plan, and `ArrowCastPlan` is that plan held; a record
cast reconciles Struct columns by ASCII-case-insensitive name, target order,
and Field null/default policy.
Readers and writers continue to exchange `pyarrow.RecordBatchReader` values;
class declarations remain schema definitions rather than a second row I/O
surface.

Record writes name both their Python representation and intent:
`overwrite_arrow_reader`, `append_arrow_table`, `merge_arrow_batch`,
and the corresponding `*_records`, pandas, and polars adapters all redirect to
the same streamed Rust pipeline. Every call takes one keyword-only `options=`
value. Set `options.commit_batch_num = N` to publish every `N` whole batches
of the stream plus the final remainder - Python rows are converted
`batch_row_size` rows to a batch, and a cadence never cuts one; leave it as
`None` for the destination's own cadence: a file or folder publishes once after
successful end of input, an Iceberg table each time the batches it holds reach
its target file size. A later failure leaves completed cadences visible by
design, and zero is rejected before a one-shot Python input is inspected.

For configured intent, the same representations expose `write_*` with the
canonical `(input, mode, *, options=None)` order. `mode` is required and is one
of `"overwrite"`, `"append"`, or `"merge"`; it is validated before the input
is exported or iterated. `is_io()` reports the general byte/record capability,
while lazy `row_size` and `column_size` read whole-media dimensions from native
format metadata and cache them only while the handle is open.

`DataType(value)` infers native wrappers, datatype expression strings,
PyArrow datatypes, and Python annotations such as `str` or `list[int]`.
`DataType.decimal(precision, scale=0)` also accepts integer-like objects and
base-10 numeric strings, selecting Decimal128 through precision 38 and
Decimal256 through precision 76.
`DataType.time(unit)` accepts the shared native unit aliases and selects
Time32 for seconds/milliseconds or Time64 for microseconds/nanoseconds.
`Scalar` exposes checked native arithmetic through `add`, `subtract`,
`multiply`, `divide`, `remainder`, `negate`, and `absolute`, plus Python's
matching operators (including reflected operators and `abs`). Python-native
operands are inferred once at the boundary; overflow, zero division, inexact
decimal division, and invalid operand kinds retain distinct Python errors.
`Expression` uses the same arithmetic names and operators to build native
trees. Strings remain expression grammar (`"price"` names a column and
`"'fee'"` is a literal); every non-string Python operand is inferred as one
native `Scalar`, and reflected operators preserve left-to-right order.
Avro schemas compare, order, and hash by their complete retained native schema
identity; `fingerprint()` remains the separate Parsing Canonical Form identity
that intentionally omits annotations. Decoded Avro containers and immutable
Iceberg metadata values use the core's complete equality, ordering, and stable
hash identities, with exact copy, repr, and pickle round-trips. Iceberg
`ScanPlan` is a self-contained five-count report, so it supports the same value
protocols without retaining a table or executable scan. `IcebergOptions` is
mutable until first hashed; hashing locks every setter, while copied and
unpickled options are independent unlocked values. Lazy Avro blocks, bound
expressions/statements, and generic `RecordOptions` deliberately remain
unhashable because the core defines no complete canonical value identity for
their operational state.
Bare `DataType.variant()` is the self-describing semi-structured Variant
datatype; `DataType.variant(fields)` stays the dense-union sugar building the
canonical dense Union with sequential native type IDs - the parenthesis
disambiguates. `yggio.variant` builds the bare Variant field,
`yggio.dense_union` the union one, and explicit `yggio.union` remains
available for custom IDs or sparse layout.

Every page of the [documentation](https://platob.github.io/yggio/) shows its
operation in Rust, Python and JavaScript, so a Python spelling is documented
where the vocabulary it belongs to is. The field classes are their own
[guide](FIELDS.md).

