Metadata-Version: 2.4
Name: schema-sanitizer
Version: 0.4.2
Summary: Path- and URI-based data sanitization for CSV, JSON, JSON arrays, JSON Lines, XML, and Parquet pipelines.
Keywords: arrow,pyarrow,json,xml,csv,schema,sanitization
Author: bgallan
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Project-URL: Homepage, https://github.com/bgallan/schema-sanitizer
Project-URL: Repository, https://github.com/bgallan/schema-sanitizer
Project-URL: Changelog, https://github.com/bgallan/schema-sanitizer/releases
Project-URL: Issues, https://github.com/bgallan/schema-sanitizer/issues
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: detect-secrets==1.5.0; extra == "dev"
Requires-Dist: ruff>=0.6.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: aiohttp>=3.9; extra == "dev"
Requires-Dist: pyarrow>=14.0.0; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Requires-Dist: polars>=0.20; extra == "dev"
Requires-Dist: duckdb>=1.0; extra == "dev"
Provides-Extra: pyarrow
Requires-Dist: pyarrow>=14.0.0; extra == "pyarrow"
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == "polars"
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.0; extra == "duckdb"
Provides-Extra: gcs
Requires-Dist: aiohttp>=3.9; extra == "gcs"
Requires-Dist: google-auth>=2.0; extra == "gcs"
Provides-Extra: s3
Requires-Dist: aiobotocore>=2.13; extra == "s3"
Requires-Dist: botocore>=1.34; extra == "s3"
Provides-Extra: azure
Requires-Dist: aiohttp>=3.9; extra == "azure"
Requires-Dist: azure-identity>=1.16; extra == "azure"
Requires-Dist: azure-storage-blob>=12.20; extra == "azure"
Provides-Extra: bigquery
Requires-Dist: pyarrow>=14.0.0; extra == "bigquery"
Requires-Dist: adbc-driver-bigquery[dbapi]; extra == "bigquery"
Provides-Extra: cloud
Requires-Dist: aiohttp>=3.9; extra == "cloud"
Requires-Dist: google-auth>=2.0; extra == "cloud"
Requires-Dist: aiobotocore>=2.13; extra == "cloud"
Requires-Dist: botocore>=1.34; extra == "cloud"
Requires-Dist: azure-identity>=1.16; extra == "cloud"
Requires-Dist: azure-storage-blob>=12.20; extra == "cloud"
Provides-Extra: all
Requires-Dist: pyarrow>=14.0.0; extra == "all"
Requires-Dist: polars>=0.20; extra == "all"
Requires-Dist: pandas>=2.0; extra == "all"
Requires-Dist: duckdb>=1.0; extra == "all"
Requires-Dist: adbc-driver-bigquery[dbapi]; extra == "all"
Requires-Dist: aiohttp>=3.9; extra == "all"
Requires-Dist: google-auth>=2.0; extra == "all"
Requires-Dist: aiobotocore>=2.13; extra == "all"
Requires-Dist: botocore>=1.34; extra == "all"
Requires-Dist: azure-identity>=1.16; extra == "all"
Requires-Dist: azure-storage-blob>=12.20; extra == "all"
Description-Content-Type: text/markdown

# schema-sanitizer

`schema-sanitizer` turns inconsistent CSV, JSON, JSON Lines, XML, and Parquet
data into analytical tables or clean files with a stable schema.

Its native C++23 engine handles reading, inference, reconciliation, and
materialization. The Python API adds PyArrow, pandas, Polars, and DuckDB outputs,
file writers, cloud access, and partitioned pipelines.

> The project is alpha software. Its primary focus is Parquet pipelines and
> BigQuery external tables.

## Index

- [Install](#install)
- [First conversion](#first-conversion)
- [What it does](#what-it-does)
- [Two ways to use the API](#two-ways-to-use-the-api)
- [Schema and memory](#schema-and-memory)
- [Documentation](#documentation)
- [Development](#development)
- [License](#license)

## [Install](#index)

Install the package with the adapter you need:

```bash
pip install "schema-sanitizer[pyarrow]"
```

Extras are also available for `polars`, `pandas`, `duckdb`, `gcs`, `s3`,
`azure`, `bigquery`, `cloud`, and `all`.

## [First conversion](#index)

```python
import schema_sanitizer as ss

result = ss.to_pyarrow(
    "raw/events.jsonl",
    input_format="jsonl",
    parse_integers=True,
    parse_iso_timestamps=True,
)

table = result.clean_data
print(table.schema)
print(result.schema_drifts)
```

Write directly to Parquet without retaining a complete table in memory:

```python
ss.to_parquet(
    "raw/events.jsonl",
    "silver/events.parquet",
    input_format="jsonl",
    multi_threading=True,
)
```

Every `to_*` conversion returns a `Result` containing the output, statistics,
execution policy, schema registry, and detected schema changes.

## [What it does](#index)

- Reads CSV, JSON, JSON arrays, JSONL/NDJSON, XML, Parquet, and Python
  dictionary iterables.
- Processes individual files, non-recursive directories, and remote objects.
- Produces PyArrow, pandas, Polars, DuckDB, CSV, JSONL, or Parquet.
- Carries a schema registry across runs and reports every schema change.
- Reconciles reordered or additive CSV headers.
- Uses one global memory limit and adapts concurrency to the machine.
- Supports local paths, GCS, S3, Azure Blob, and HTTP(S) files.
- Builds Parquet pipelines from Hive partitions or object modification times.
- Generates and maintains BigQuery external tables and schema sidecars.

## [Two ways to use the API](#index)

The `to_*` functions are convenient for one-off calls. Create a `Sanitizer` to
reuse one configuration:

```python
sanitizer = ss.Sanitizer(
    ss.SanitizeOptions(
        input_format="csv",
        csv=ss.CsvOptions(header_mode="union"),
        parsing=ss.ParsingOptions(iso_dates=True),
        resources=ss.ResourceOptions(
            multi_threading=True,
            memory_limit_bytes=512 * 1024 * 1024,
        ),
    )
)

frame = sanitizer.to_polars("raw/daily/").clean_data
```

## [Schema and memory](#index)

Pass one run's registry to the next to evolve the schema deterministically:

```python
first = ss.to_parquet(
    "raw/day-1.jsonl",
    "silver/day-1.parquet",
    input_format="jsonl",
)
second = ss.to_parquet(
    "raw/day-2.jsonl",
    "silver/day-2.parquet",
    input_format="jsonl",
    schema_registry=first.schema_registry,
)
```

`memory_limit_bytes` bounds resources owned by the conversion. Streaming readers
and writers can process files larger than that budget. A returned analytical
table or DataFrame becomes caller-owned and is outside the budget; direct file
output is the safe choice when a complete result may be too large.

## [Documentation](#index)

The [documentation guide](docs/README.md) organizes detailed material by task:

- [Getting started](docs/guides/getting-started.md)
- [Python API](docs/reference/python-api.md)
- [Configuration options](docs/reference/options.md)
- [Inputs and filesystems](docs/reference/inputs-and-filesystems.md)
- [Schemas and registries](docs/reference/schema-and-registry.md)
- [Partitioned pipelines](docs/guides/partitioned-pipelines.md)
- [BigQuery integration](docs/reference/bigquery.md)
- [Resources and concurrency](docs/operations/resources-and-concurrency.md)
- [Reader security](docs/operations/reader-security-limits.md)
- [Compatibility](docs/reference/compatibility.md)
- [CI/CD pipeline](docs/project/ci-cd.md)
- [Development and contribution](docs/project/development.md)

Complete programs live in [`examples/`](examples/).
[Example 7](examples/example_07/07_gcs_jsonl_to_silver_parquet_range_prefix.py)
covers a Hive pipeline to Parquet and BigQuery.
[Example 8](examples/example_08/08_gcs_csv_modified_window_to_polars_parquet.py)
covers CSV under a flat GCS prefix, modification-time windows, a custom Polars
transformation, and UTC `year/month/day` Hive output from a chosen timestamp.

## [Development](#index)

```bash
python -m pip install -e ".[dev]"
pytest -q
pre-commit run --all-files
```

See the [development guide](docs/project/development.md) for native builds, focused
tests, benchmarks, and CI.

## [License](#index)

Apache License 2.0. See [LICENSE](LICENSE).
