Metadata-Version: 2.4
Name: chooong.datasift
Version: 0.1.0
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Environment :: Console
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Rust
Classifier: Topic :: Office/Business
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Filters
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Dist: pydantic>=2.12,<3
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.txt
License-File: licenses/third-party/018b1cb87efdf7a04c2fcc13d57ed63f62149113fb207b27ea13430d64f13513.txt
License-File: licenses/third-party/01c266bced4a434da0051174d6bee16a4c82cf634e2679b6155d40d75012390f.txt
License-File: licenses/third-party/0218327e7a480793ffdd4eb792379a9709e5c135c7ba267f709d6f6d4d70af0a.txt
License-File: licenses/third-party/025436edff4cfcdde17a5811fdea78892d8482efd1abdec5a17872d07a4f2112.txt
License-File: licenses/third-party/03548525ec3706de58e48ef6e3c455582557a0ad3b54a2da9a3819add35dfb72.txt
License-File: licenses/third-party/0444c6991eead6822f7b9102e654448d51624431119546492e8b231db42c48bb.txt
License-File: licenses/third-party/074e6e32c86a4c0ef8b3ed25b721ca23aca83df277cd88106ef7177c354615ff.txt
License-File: licenses/third-party/0cec06e0e55fbc3dc5cee4fca9b607f66cb8f4e4dbcf3b3c013594dd156732e9.txt
License-File: licenses/third-party/0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f.txt
License-File: licenses/third-party/10c8f8922ec0a74e8721148bfc5b5e18c4f989d62e38f5580a501a3e3c1d0391.txt
License-File: licenses/third-party/11789f45bb180841cd362a5eee6789c68ddb573a11105e30768c308a6add0190.txt
License-File: licenses/third-party/123a331b5dbf04c30097fa43b8f858bc85df671fe776de498d01f3d6b7c1f69e.txt
License-File: licenses/third-party/172020dbfd5b53a226dfde77616190a48dcff519b0bc0e6deb91a8450782c4af.txt
License-File: licenses/third-party/1847e0e0698142ed4347c1441a9fa81c8fbddd44b1d8bbcd5e3647f991759d7f.txt
License-File: licenses/third-party/1a2f5c12ddc934d58956aa5dbdd3255fe55fd957633ab7d0d39e4f0daa73f7df.txt
License-File: licenses/third-party/209fbbe0ad52d9235e37badf9cadfe4dbdc87203179c0899e738b39ade42177b.txt
License-File: licenses/third-party/23860c2a7b5d96b21569afedf033469bab9fe14a1b24a35068b8641c578ce24d.txt
License-File: licenses/third-party/23f18e03dc49df91622fe2a76176497404e46ced8a715d9d2b67a7446571cca3.txt
License-File: licenses/third-party/268872b9816f90fd8e85db5a28d33f8150ebb8dd016653fb39ef1f94f2686bc5.txt
License-File: licenses/third-party/29e9fe5074bd27e0e5d5d110394fbbcd841baee2651a3c4b4560a632702cede4.txt
License-File: licenses/third-party/3290ae0fbc9ddb77d2239121d710f0bb9d31b3b4744e6d97fe01e652b4c1870b.txt
License-File: licenses/third-party/32c76dbe0e73d79100d5ece77c158399f2e2541bc5c78548a4ba45c1cb53c5c9.txt
License-File: licenses/third-party/35242e7a83f69875e6edeff02291e688c97caafe2f8902e4e19b49d3e78b4cab.txt
License-File: licenses/third-party/377c2e7c53250cc5905c0b0532d35973392af16ffb9596a41d99d202cf3617c9.txt
License-File: licenses/third-party/378f5840b258e2779c39418f3f2d7b2ba96f1c7917dd6be0713f88305dbda397.txt
License-File: licenses/third-party/3823dda7cf046602f4b4e77ec8e227863dc4736037cc85bb33d9f19febe16bb7.txt
License-File: licenses/third-party/3fa4ca83dcc9237839b1bdeb2e6d16bdfb5ec0c5ce42b24694d8bbf0dcbef72c.txt
License-File: licenses/third-party/42a35170233e83e18856792e748de4c1ce4a63b2afce9a370c89ef3fe23f9f2d.txt
License-File: licenses/third-party/42fa16951ce7f24b5a467a40e5b449a1d41e662f97ca779864f053f39e097737.txt
License-File: licenses/third-party/436bc5a105d8e57dcd8778730f3754f7bf39c14d2f530e4cde4bd2d17a83ec3d.txt
License-File: licenses/third-party/45f522cacecb1023856e46df79ca625dfc550c94910078bd8aec6e02880b3d42.txt
License-File: licenses/third-party/47dc9ff29128ddfb4d6a0435383c9f89120bc374dbcc1dd00b933a0b28aa7865.txt
License-File: licenses/third-party/495c30b45120f8af07cfa26eb9cb1ebfe8324560dca2b3b1e76cad1b9b6b489a.txt
License-File: licenses/third-party/4cada0bd02ea3692eee6f16400d86c6508bbd3bafb2b65fed0419f36d4f83e8f.txt
License-File: licenses/third-party/4da95ec4ecb65b738d470b7d762894ad9c97da93e6cbfb18b570fc2c96f4b871.txt
License-File: licenses/third-party/508a77d2e7b51d98adeed32648ad124b7b30241a8e70b2e72c99f92d8e5874d1.txt
License-File: licenses/third-party/516b24e051bf5630880ebbd55c40a25ce9552ebaf8970a53e8976eb70e522406.txt
License-File: licenses/third-party/523a42c25d245dde9c015f882cec7f4555aad883382a6cf19b4b7d9b2cd5419b.txt
License-File: licenses/third-party/58545fed1565e42d687aecec6897d35c6d37ccb71479a137c0deb2203e125c79.txt
License-File: licenses/third-party/5a93d5831e1297ab10fe643e1a631e83be392896da14ee2951285a79012df69d.txt
License-File: licenses/third-party/5b3d8a674979e158328c57f5c4cc12150df79d0bdf12c86eafc2e9c6006a2d4e.txt
License-File: licenses/third-party/61d383b05b87d78f94d2937e2580cce47226d17823c0430fbcad09596537efcf.txt
License-File: licenses/third-party/62065228e42caebca7e7d7db1204cbb867033de5982ca4009928915e4095f3a3.txt
License-File: licenses/third-party/62c7a1e35f56406896d7aa7ca52d0cc0d272ac022b5d2796e7d6905db8a3636a.txt
License-File: licenses/third-party/6485b8ed310d3f0340bf1ad1f47645069ce4069dcc6bb46c7d5c6faf41de1fdb.txt
License-File: licenses/third-party/65f94e99ddaf4f5d1782a6dae23f35d4293a9a01444a13135a6887017d353cee.txt
License-File: licenses/third-party/66e3ee1fa7f909ad3c612d556f2a0cdabcd809ad6e66f3b0605015ac64841b70.txt
License-File: licenses/third-party/696759d65dfe558ff7d9f031c76db19ec5c0767470fb67c4e8d990820d1e99c9.txt
License-File: licenses/third-party/6b1691fdee3b03c3946696a7be125fcf50eff310b9486fac850c1b8aea91478b.txt
License-File: licenses/third-party/6df43f6f4b5d4587f3d8d71e45532c688fd168afa5fe89d571cb32fa09c4ef51.txt
License-File: licenses/third-party/70c9d40f1f9545c3f133b8a67206e89da850f6468eed072281bb3701514114a9.txt
License-File: licenses/third-party/7365cc8878a1d7ce155a58c4ca09c3d7a6be413efa5334a80ea842912b669349.txt
License-File: licenses/third-party/73b9dc2e79c7308998dd30296e073aefaefb944a68fb89aa412c23c0edcabcaa.txt
License-File: licenses/third-party/74db5baf44a41b1000312c673544b3374e4198af5605c7f9080a402cec42cfa3.txt
License-File: licenses/third-party/769f80b5bcb42ed0af4e4d2fd74e1ac9bf843cb80c5a29219d1ef3544428a6bb.txt
License-File: licenses/third-party/7b63ecd5f1902af1b63729947373683c32745c16a10e8e6292e2e2dcd7e90ae0.txt
License-File: licenses/third-party/7b724c8fa4a82af733caccabdaa21e2bf8bcb0822fe62d5cf03f99d458438705.txt
License-File: licenses/third-party/7ca1297d23644e30bd489193a82a33f324e5fe33f25df4195649b91b883df967.txt
License-File: licenses/third-party/831e0f43ad0bf014c1c4fec5767aae470434c1d66d6e671be2d823e729063e25.txt
License-File: licenses/third-party/838118388fe5c2e7f1dbbaeed13e1c7f3ebf88be91319c7c1d77c18e987d1a50.txt
License-File: licenses/third-party/83c1763356e822adde0a2cae748d938a73fdc263849ccff6b27776dff213bd32.txt
License-File: licenses/third-party/84b34dd7608f7fb9b17bd588a6bf392bf7de504e2716f024a77d89f1b145a151.txt
License-File: licenses/third-party/8569ebf6bf52937b961d9c9c934d8400a2bb6b09651a9afacbb3fc18fe5c09b1.txt
License-File: licenses/third-party/8764a597675778ddfd4e25f81b08a05dbcf089ac05662df7613fe67f150e3aa2.txt
License-File: licenses/third-party/87d9feb9238c6bd8e0024fc4733b06cff036f89f36d93b7df1c8a0549bbb7a5b.txt
License-File: licenses/third-party/90567e2718bf7fd65a71a3a43c5596488e80e5f51ed02bfea6fec54458b5f3d1.txt
License-File: licenses/third-party/90eb64f0279b0d9432accfa6023ff803bc4965212383697eee27a0f426d5f8d5.txt
License-File: licenses/third-party/946c9835d8034d24404f8cfec5f4654cee5dad17e944afc3d06d742cf2882831.txt
License-File: licenses/third-party/98181e7249d0c01737645ec982499ce99a0f07eb8f7d625b8840d799d10dbc01.txt
License-File: licenses/third-party/9d185ac6703c4b0453974c0d85e9eee43e6941009296bb1f5eb0b54e2329e9f3.txt
License-File: licenses/third-party/a01a418b6ffb1de8f4c3814e386b35e1a93a14529b68f17d286550f355b298e1.txt
License-File: licenses/third-party/a60eea817514531668d7e00765731449fe14d059d3249e0bc93b36de45f759f2.txt
License-File: licenses/third-party/a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9.txt
License-File: licenses/third-party/a825bd853ab71619a4923d7b4311221427848070ff44d990da39b0b274c1683f.txt
License-File: licenses/third-party/a8799c2a07dabcb8d52481f885c5ab7000acb8a4d3cd9503fbcb97c84d350e65.txt
License-File: licenses/third-party/a9040321c3712d8fd0b09cf52b17445de04a23a10165049ae187cd39e5c86be5.txt
License-File: licenses/third-party/aaff376532ea30a0cd5330b9502ad4a4c8bf769c539c87ffe78819d188a18ebf.txt
License-File: licenses/third-party/ab499c75a0f0da8e0fe83bf9ba3a27e5c39705310f07107c6c66b69958d2401c.txt
License-File: licenses/third-party/aecc100d0547aa3bcbe04bcf5cb87c7db0be11004602adfa5056edf8bb847b35.txt
License-File: licenses/third-party/af59cea35d7f5e2777a713b8d155d65efa2c339eb43f3c14e868c6ac8506edad.txt
License-File: licenses/third-party/afcbe3b2e6b37172b5a9ca869ee4c0b8cdc09316e5d4384864154482c33e5af6.txt
License-File: licenses/third-party/b05785f9f18e6716bab63424b11454513b9943a222595b70411009202fc592b5.txt
License-File: licenses/third-party/b1181a40b2a7b25cf66fd01481713bc1005df082c53ef73e851e55071b102744.txt
License-File: licenses/third-party/b71bd43a069ca0641a9ecfe585ca7b3c53b5cc1608f8b68321168698e28b5ea1.txt
License-File: licenses/third-party/bfb1112d49db5b1daecdfef24bd7e2f3ea0bafb33aa67aa0ab51e2bf8407c03d.txt
License-File: licenses/third-party/c16f8dcf1a368b83be78d826ea23de4079fe1b4469a0ab9ee20563f37ff3d44b.txt
License-File: licenses/third-party/c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383.txt
License-File: licenses/third-party/c6596eb7be8581c18be736c846fb9173b69eccf6ef94c5135893ec56bd92ba08.txt
License-File: licenses/third-party/c9bff75738922193e67fa726fa225535870d2aa1059f91452c411736284ad566.txt
License-File: licenses/third-party/cc8cc7f31c5f26b85702a161ba09bbe8b77a473c370709939e89db848b9d1f30.txt
License-File: licenses/third-party/cdea3888d561fddcce138060983f9e78c81fc7927a50938303508a8d254ee8fc.txt
License-File: licenses/third-party/cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30.txt
License-File: licenses/third-party/d00d6d48df2db6f91e4ca9c656e5e30782d5becf52bb3e5bfd0aff467bd37781.txt
License-File: licenses/third-party/d2e7ec5355c96eeade56b09187ceb48a6a30299da3ce7531a66d3d11405ab963.txt
License-File: licenses/third-party/d415a86ccfd794129666247cbbadebee7a6d0ac3ff701c76c07b0f9424f0307d.txt
License-File: licenses/third-party/da28ccc6b158fc2d8cccc74e99794b1cff1d29bd7bbeb019442fcf0c04c6cad9.txt
License-File: licenses/third-party/db11fec9946737df39ca3898d9cd8c10ec6f6c3a884a6802b0ad0b81b4e8f23a.txt
License-File: licenses/third-party/e196ffeb5ab101e9ec8bbe395c9f367c5380ee2d28c47e5c03371523798a9af7.txt
License-File: licenses/third-party/e34c58338bd89d43e709e226610d8f32b3e3c47f4ad9a99a8dc1d4ac7842488e.txt
License-File: licenses/third-party/e5b198d7f809311723889814d99a92f808c69e89a847efa7a7b65270f5fdeae6.txt
License-File: licenses/third-party/e72111c52b7d96ebe25348dee19f0744f444d3c95ae6b1ecb6ccaecc5bce05ba.txt
License-File: licenses/third-party/ecc269ef87fd38a1d98e30bfac9ba964a9dbd9315c3770fed98d4d7cb5882055.txt
License-File: licenses/third-party/ef2b3bbd7b718a7881253226313f103e89b379e1279681a3c1ebcb65a27d209d.txt
License-File: licenses/third-party/f0c1d22de8bd3b776a40f2691f08b84e91c7bafc78b72a51150e231002b88db3.txt
License-File: licenses/third-party/f0cf9b1c62bbe3bd3a69f5f79c7158f513f612b4940a0a812d1db39d605318bc.txt
License-File: licenses/third-party/f59bcb1c02654665f78a5f290bae76b6358137675f6459f51e6d474a4623d8ea.txt
License-File: licenses/third-party/f7db81051789b729fea528a63ec4c938fdcb93d9d61d97dc8cc2e9df6d47f2a1.txt
License-File: licenses/third-party/f8218253704e32441cafea1b9b3bcb2c6a3c51c5553cd8513d179290202bccb2.txt
License-File: licenses/third-party/fb77f0a9c53e473abe5103c8632ef9f0f2874d4fb3f17cb2d8c661aab9cee9d7.txt
License-File: licenses/third-party/fd80a26fbb3f644af1fa994134446702932968519797227e07a1368dea80f0bc.txt
License-File: licenses/third-party/ff8f68cb076caf8cefe7a6430d4ac086ce6af2ca8ce2c4e5a2004d4552ef52a2.txt
Summary: Native streaming Excel/CSV discovery, validation, filtering and NDJSON/Parquet export
Keywords: excel,xlsx,xlsm,csv,rust,streaming,table-discovery,data-extraction,data-validation,data-filtering,data-processing,etl,parquet,ndjson
Home-Page: https://github.com/chooong-group/chooong.datasift
Author: Marcus Steinbach
Maintainer: Chooong
License-Expression: Apache-2.0
Requires-Python: >=3.12, <3.15
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/chooong-group/chooong.datasift/blob/main/CHANGELOG.md
Project-URL: Documentation, https://datasift.chooong.com
Project-URL: Homepage, https://github.com/chooong-group/chooong.datasift
Project-URL: Issues, https://github.com/chooong-group/chooong.datasift/issues
Project-URL: Repository, https://github.com/chooong-group/chooong.datasift

# chooong.datasift

[![CI][ci-badge]][ci]
[![PyPI version][pypi-badge]][pypi]
[![Python versions][python-badge]][installation]
[![License: Apache-2.0][license-badge]][license]

[ci-badge]: https://github.com/chooong-group/chooong.datasift/actions/workflows/ci.yml/badge.svg
[ci]: https://github.com/chooong-group/chooong.datasift/actions/workflows/ci.yml
[pypi-badge]: https://img.shields.io/pypi/v/chooong.datasift.svg
[pypi]: https://pypi.org/project/chooong.datasift/
[python-badge]: https://img.shields.io/badge/python-3.12%20%7C%203.13%20%7C%203.14-blue
[license-badge]: https://img.shields.io/badge/license-Apache--2.0-blue
[installation]: https://github.com/chooong-group/chooong.datasift/blob/main/docs/installation.md
[license]: https://github.com/chooong-group/chooong.datasift/blob/main/LICENSE

Extract, validate and filter tables from XLSX/XLSM and CSV with a JSON mask.
Rust streams rows and writes NDJSON/Parquet; Python exposes typed, immutable
results. Automatic table discovery finds headers away from A1 and multiple table
regions in Excel and CSV without fixed coordinates in the mask. Start with the
[discovery guide][discovery-guide] for Named-column and Free Discovery examples.

[discovery-guide]: https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/automatic-table-discovery.md

## Installation

Requires CPython 3.12–3.14.

<!-- check: syntax -->

```bash
python -m pip install chooong.datasift
```

Compatible prebuilt wheels do not require a Rust compiler.
Building from source requires Rust, Cargo and a native linker.

See the [installation guide][installation] for details.
Pydantic is the only direct Python runtime dependency; no external service is needed.

## Features

### Input formats

- Read **XLSX, XLSM and CSV** files without loading entire worksheets into a DataFrame.
- Process multiple files and wildcard paths in deterministic order.
- Scan individual Excel worksheets or all worksheets in a workbook.
- Configure CSV encodings, delimiters and quoting, including multiline fields.
- Read cached Excel formula results without executing formulas or macros.

### Table discovery

- Automatically discover tables that do not start at A1.
- Detect multiple tables within a worksheet, including adjacent and vertically separated tables.
- Join a configured number of physical header rows, recognize repeated headers and optionally fill
  actual Excel merged ranges; [discovery limits][discovery-limits] depend on the mode.
- Normalize column names with trimming, case conversion, Unicode normalization, whitespace handling
  and aliases.
- Resolve duplicate column names with deterministic suffixes.
- Locate tables using header anchors independently of filter conditions.
- Define explicit table sources with column mappings, row ranges and excluded rows.
- Apply independent rules to different table sources.
- Detect ambiguous layouts instead of silently guessing table boundaries.

Named-column Discovery uses anchors or filter column names, including for text-only data.
Free Discovery needs scalar evidence immediately after each candidate header.
Comments and totals require explicit row rules or predicates; their meaning is not inferred.
CSV table discovery does not sniff delimiters or encodings.

[discovery-limits]: https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/complex-layouts.md#supported-layouts-and-limits

### Filtering and column selection

- Define filters using JSON or typed Python models.
- Combine conditions with **AND, OR and NOT**.
- Compare values using equality, inequality, ordering and list membership.
- Filter text using substring, prefix, suffix and regular-expression matching.
- Check for empty values and specific data types.
- Select output columns independently of filter conditions.
- Select columns by name, wildcard or regular expression.
- Control output column order without modifying the source.
- Filter and select data directly in the native Rust engine.

### Validation and data types

- Enforce **required fields** and reject records with missing values.
- Replace missing values with configurable defaults.
- Validate column types and optionally convert values.
- Support strings, integers, floats, booleans, Decimal, money, dates and datetimes.
- Parse localized numeric and monetary values.
- Preserve exact decimal and monetary values without implicit rounding.
- Handle Excel date systems, ISO dates and timezone-aware datetimes.
- Apply validation independently of output column selection.
- Preserve original values and detailed rejection reasons for invalid records.

### Row processing and streaming

- Classify rows as data, headers, repeated headers, empty rows, comments, totals or separators.
- Include or exclude row classes using configurable rules.
- Stream results in bounded batches with backpressure.
- Iterate over matching rows or receive matched and rejected records in a single ordered stream.
- Return immutable, typed Python row objects.
- Support early cancellation and explicit iterator cleanup.
- Report row counts, conversions, defaults, rejections and discovered table schemas.

### Export and reliability

- Export matching and rejected records to **NDJSON or Parquet**.
- Write output directly from Rust without Python-side row serialization.
- Preserve source metadata in rejected NDJSON and both Parquet outputs; matched NDJSON contains
  projected values only. Rejected outputs also retain rejection details.
- Preserve existing output files when scanning or serialization fails.
- Use journaled commits and verified recovery for interrupted file replacements.
- Continue processing independent tables after schema errors.
- Optionally continue processing remaining input files after file errors while reporting incomplete results.
- Enforce documented parser, schema, expression and resource limits.

### Python API and CLI

- Use the same filter configuration through Python and the command-line interface.
- Validate JSON masks with Pydantic and generate JSON Schema.
- Access matching rows, batches, detailed scan records and scan statistics.
- Inspect discovered tables, column bindings and diagnostics.
- Use CLI output files, stdout streaming and JSON execution reports.
- Support CPython **3.12, 3.13 and 3.14** with a native Rust extension.

## Example

This example creates its own input:

```python
from pathlib import Path

from chooong.datasift import (
    AllOf,
    DataType,
    FilterMask,
    HeaderConfig,
    NumericCondition,
    Operator,
    ScalarCondition,
    TypeRule,
    iter_matching_rows,
)

Path("transactions.csv").write_text(
    "Country,Amount,Note\nDE,1500,first\nAT,2000,second\nDE,900,third\n",
    encoding="utf-8",
)

mask = FilterMask(
    sheet=None,
    header=HeaderConfig(anchors=("Country", "Amount")),
    types={"Amount": TypeRule(type=DataType.INTEGER, convert=True)},
    where=AllOf(conditions=(
        ScalarCondition(column="Country", op=Operator.EQ, value="DE"),
        NumericCondition(column="Amount", op=Operator.GTE, value=1000),
    )),
)

rows = [dict(row) for row in iter_matching_rows("transactions.csv", mask)]
assert rows == [{"Country": "DE", "Amount": 1500}]

Path("mask.json").write_text(
    mask.model_dump_json(by_alias=True),
    encoding="utf-8",
)
```

Run the same filter from the CLI:

```bash
chooong-datasift transactions.csv mask.json --output matches.ndjson
```

## Column selection and validation

DataSift separates row filtering, column selection and validation. Use `select` to choose output
columns, `types` to convert values and `required` to reject records with missing values.

```python
from pathlib import Path

from chooong.datasift import (
    DataType,
    FilterMask,
    HeaderConfig,
    NumericCondition,
    Operator,
    TypeRule,
    iter_matching_rows,
)

Path("orders.csv").write_text(
    "Country,Amount,Reference\n"
    "DE,1500,INV-001\n"
    "DE,2200,\n"
    "AT,1900,INV-003\n"
    "DE,900,INV-004\n",
    encoding="utf-8",
)

mask = FilterMask(
    sheet=None,
    header=HeaderConfig(anchors=("Country", "Amount", "Reference")),
    select=("Reference", "Amount"),
    required=("Reference",),
    types={
        "Amount": TypeRule(type=DataType.INTEGER, convert=True),
    },
    where=NumericCondition(
        column="Amount",
        op=Operator.GTE,
        value=1000,
    ),
)

rows = [dict(row) for row in iter_matching_rows("orders.csv", mask)]

assert rows == [
    {"Reference": "INV-001", "Amount": 1500},
    {"Reference": "INV-003", "Amount": 1900},
]
```

The mask applies three independent operations:

- **Column selection:** `select` returns only `Reference` and `Amount`, in that order. `Country` is excluded.
- **Type conversion:** `types` converts CSV text in `Amount` to integers before numeric comparison.
- **Required fields:** `required` rejects records where `Reference` is missing or empty.

The `where` condition matches amounts greater than or equal to 1000.

The record with amount 2200 is rejected because its reference is missing. The record with amount 900
does not match the filter. Neither appears in the matching results.

Rejected records remain available through `scan_batches`, including their original values and
rejection reasons.

See the [filter reference][filter-reference] for the complete configuration and the
[validation guide][validation-guide] for handling rejected records.

[filter-reference]: https://github.com/chooong-group/chooong.datasift/blob/main/docs/api/filters.md
[validation-guide]: https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/validation.md

## Documentation and limits

Start with [installation](https://github.com/chooong-group/chooong.datasift/blob/main/docs/installation.md)
and the [quickstart](https://github.com/chooong-group/chooong.datasift/blob/main/docs/quickstart.md).
The [documentation](https://github.com/chooong-group/chooong.datasift/blob/main/docs/index.md)
links guides, exact Python/CLI references and the filter schema. Practical workflows:

- [JSON filters](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/advanced-filtering.md)
  and [column selection](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/column-selection.md).
- [Per-table rules](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/complex-layouts.md#different-rules-per-table)
  and [inspection/diagnostics](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/table-inspection.md).
- [Batch processing](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/batch-processing.md)
  and [partial results](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/error-handling.md).
- [Export formats](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/exporting.md)
  and [resource use](https://github.com/chooong-group/chooong.datasift/blob/main/docs/guides/performance.md).

XLS/XLSB, encrypted workbooks, formula calculation and Excel output are unsupported.
Formula caches are read. Shared strings and metadata use memory despite row
streaming. Ambiguous layouts require anchors or explicit sources. A late scan
failure may follow partial iterator/stdout results; exported results use a
journaled commit, not a shared atomic two-file replacement. See
[file safety](https://github.com/chooong-group/chooong.datasift/blob/main/docs/reference/file-safety.md).

Public imports come from `chooong.datasift`. Python validates masks; the
[Rust core](https://github.com/chooong-group/chooong.datasift/blob/main/datasift-core/README.md)
owns parsing and the row pipeline. For development, use `uv sync --frozen` and
[Contributing](https://github.com/chooong-group/chooong.datasift/blob/main/CONTRIBUTING.md).

[Changelog](https://github.com/chooong-group/chooong.datasift/blob/main/CHANGELOG.md) ·
[Security](https://github.com/chooong-group/chooong.datasift/blob/main/SECURITY.md) ·
[Apache-2.0 license](https://github.com/chooong-group/chooong.datasift/blob/main/LICENSE) ·
[Support development](https://github.com/sponsors/marcus-steinbach)

