Metadata-Version: 2.4
Name: winnow-md
Version: 0.1.0
Summary: Remove boilerplate from scraped markdown before it reaches an LLM. Subtractive-only, auditable, zero dependencies.
Author: Esayas Beshah
License: MIT
Keywords: markdown,boilerplate,llm,scraping,cleaning,tokens
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tokens
Requires-Dist: tiktoken; extra == "tokens"
Provides-Extra: model
Requires-Dist: numpy; extra == "model"
Requires-Dist: model2vec; extra == "model"
Dynamic: license-file

# Winnow

**Remove boilerplate from scraped markdown before it reaches an LLM.**

Scraping and extract APIs like Tavily hand
you "LLM-ready" markdown that still carries navigation, footers, cookie
banners, promos, and link farms — 10–90% of the tokens depending on the page.
The same goes for your own scraper or loader pipeline: if it produces
markdown (or HTML converted to markdown), Winnow slots in right after it —
any markdown in, leaner markdown out, with a receipt for every removed block.

- **Subtractive-only.** Winnow deletes blocks; it never rewrites a word. Zero
  hallucination risk by construction.
- **Recall-first.** Dropping real content is data loss; keeping boilerplate just
  costs tokens. When uncertain, Winnow keeps.
- **Auditable.** Every removed block comes back with a reason code and score.
- **Template memory.** Feed Winnow multiple pages from one site and it learns the
  site's template — blocks repeated across pages are boilerplate, near-certainly.
- **Zero runtime dependencies.** `pip install winnow-md` adds nothing else.

## Quickstart

```python
import winnow

# One page, heuristics only
res = winnow.clean(markdown_text)
print(res.markdown)          # cleaned markdown
print(res.stats)             # tokens before/after, reduction %
for r in res.removed:        # the receipt
    print(r.reasons, r.text[:60])

# A crawl — template memory kicks in across pages of the same domain
w = winnow.Winnow(aggressiveness=0.5)
results = w.clean_many(pages)          # list of markdown strings (or (md, url) tuples)

# Streaming with a persistent per-domain template store
w = winnow.Winnow(store="winnow.db")
res = w.clean(md, url="https://example.com/post/1")
```

```bash
winnow clean page.md                     # cleaned markdown to stdout
winnow clean ./crawl/ --report out.html  # batch + side-by-side audit report
```

Jina Reader output (`Title:` / `URL Source:` preamble) is auto-detected and the
source URL is used for template memory.

## Benchmark

Five generations of independently-labeled, adversarially-arbitrated exam
batches (each fetched fresh, dual-labeled blind, disputes refereed) — ~12,000
hand-adjudicated blocks across 21 domains:

| Exam batch | Content recall | Junk recall | Token cut |
|---|---|---|---|
| batch 5 (newest, still converging) | 0.956 | 0.61 | −42% |
| batch 4 | 0.979 | 0.59 | −49% |
| batch 3 | 0.992 | 0.56 | −35% |
| batch 2 | 0.997 | 0.55 | −30% |
| batch 1 | 1.000 | 0.66 | −41% |

Add `--url-mode strip` for roughly 15 additional points of token cut with zero
text loss. `use_model=True` (extra: `pip install winnow-md[model]`) adds a
learned block-sequence scorer that raises junk recall further and is
structurally incapable of deleting a block on its own.

The benchmark harness, labeling pipeline, and mutation self-test live in
`bench/` — see `ARCHITECTURE.md`.
