Metadata-Version: 2.4
Name: milkwashr
Version: 1.0.1
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Topic :: Text Processing
Summary: Deterministic sentence-chunk assembly (text-splitter) for the milkwash pipeline
Keywords: chunking,sentences,text-splitter,llm,rust
Home-Page: https://code.twardoch.com/milkwash/
Author-email: Adam Twardoch <adam+github@twardoch.com>
License: Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/twardoch/milkwash/blob/main/CHANGELOG.md
Project-URL: Homepage, https://github.com/twardoch/milkwash
Project-URL: Issues, https://github.com/twardoch/milkwash/issues
Project-URL: Repository, https://github.com/twardoch/milkwash.git

# milkwashr

Deterministic sentence-chunk assembly for the [milkwash](https://github.com/twardoch/milkwash)
text-laundering pipeline. Rust core, importable from Python as `milkwashr`.

Sentence *segmentation* happens upstream in Python (`sentencex`, which has
explicit Arabic and Polish support). This crate does *chunk assembly*: group
2–8 sentences per chunk with a seeded RNG, then apply
[`text-splitter`](https://crates.io/crates/text-splitter) as a size guard so an
over-long chunk cannot overflow a small model's context window.

The RNG is seeded per source text, so every model in a sweep replays the
identical chunk plan — without that, no two models see the same input and the
outputs are not comparable.

## Python

```python
import milkwashr

milkwashr.ping()                                   # "milkwashr-ok"
milkwashr.plan_chunk_spans(42, seed=1234)          # [(0, 5), (5, 9), ...]
milkwashr.assemble_chunks(sentences, seed=1234)    # list[str]
milkwashr.split_by_size(text, max_chars=6000)      # list[str]
```

## Rust

```rust
let spans = milkwashr::plan_chunk_spans(42, 1234, 2, 8);
```

## License

Apache-2.0

