Metadata-Version: 2.4
Name: winnow-md
Version: 0.1.1
Summary: Remove boilerplate from scraped markdown before it reaches an LLM. Subtractive-only, auditable, zero dependencies.
Author: Esayas Beshah
License: MIT
Project-URL: Homepage, https://github.com/Isa1asN/winnow-md
Project-URL: Repository, https://github.com/Isa1asN/winnow-md
Project-URL: Issues, https://github.com/Isa1asN/winnow-md/issues
Keywords: markdown,boilerplate,llm,scraping,cleaning,tokens
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tokens
Requires-Dist: tiktoken; extra == "tokens"
Provides-Extra: model
Requires-Dist: numpy; extra == "model"
Requires-Dist: model2vec; extra == "model"
Dynamic: license-file

# Winnow

[![PyPI](https://img.shields.io/pypi/v/winnow-md.svg)](https://pypi.org/project/winnow-md/)
[![Python](https://img.shields.io/pypi/pyversions/winnow-md.svg)](https://pypi.org/project/winnow-md/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

**Remove boilerplate from scraped markdown before it reaches an LLM.**

Scraping and extract APIs like Tavily hand
you "LLM-ready" markdown that still carries navigation, footers, cookie
banners, promos, and link farms — 10–90% of the tokens depending on the page.
The same goes for your own scraper or loader pipeline: if it produces
markdown (or HTML converted to markdown), Winnow slots in right after it —
any markdown in, leaner markdown out, with a receipt for every removed block.

- **Subtractive-only.** Winnow deletes blocks; it never rewrites a word. Zero
  hallucination risk by construction.
- **Recall-first.** Dropping real content is data loss; keeping boilerplate just
  costs tokens. When uncertain, Winnow keeps.
- **Auditable.** Every removed block comes back with a reason code and score.
- **Template memory.** Feed Winnow multiple pages from one site and it learns the
  site's template — blocks repeated across pages are boilerplate, near-certainly.
- **Zero runtime dependencies.** The core install adds nothing to your tree.

## Install

```bash
pip install winnow-md
```

Optional extras:

```bash
pip install "winnow-md[model]"    # learned block-sequence scorer (numpy + model2vec)
pip install "winnow-md[tokens]"   # exact token counts via tiktoken
pip install "winnow-md[model,tokens]"
```

Python 3.9+. The package installs as `winnow-md` and imports as `winnow`. The
core is pure Python with no dependencies; if the `[model]` extra isn't
installed, the learned scorer is silently skipped and the heuristics run alone.

## Quickstart

```python
import winnow

# One page
res = winnow.clean(markdown_text)
print(res.markdown)          # cleaned markdown
print(res.stats)             # tokens before/after, reduction %
for r in res.removed:        # the receipt
    print(r.reasons, r.text[:60])

# A crawl — template memory kicks in across pages of the same domain
w = winnow.Winnow(aggressiveness=0.5)
results = w.clean_many(pages)          # list of markdown strings (or (md, url) tuples)

# Streaming with a persistent per-domain template store
w = winnow.Winnow(store="winnow.db")
res = w.clean(md, url="https://example.com/post/1")
```

```bash
winnow clean page.md                      # cleaned markdown to stdout
winnow clean ./crawl/ --report out.html   # batch + filterable HTML audit report
winnow clean ./crawl/ --url-mode strip    # also strip URL bodies from kept links
```

Jina Reader output (`Title:` / `URL Source:` preamble) is auto-detected and the
source URL is used for template memory.

## Benchmark

Five generations of independently-labeled, adversarially-arbitrated exam
batches (each fetched fresh, dual-labeled blind, disputes refereed) — ~12,000
hand-adjudicated blocks across 21 domains:

| Exam batch | Content recall | Junk recall | Token cut |
|---|---|---|---|
| batch 5 (newest, still converging) | 0.960 | 0.61 | −42% |
| batch 4 | 0.979 | 0.61 | −49% |
| batch 3 | 0.992 | 0.57 | −35% |
| batch 2 | 0.997 | 0.58 | −29% |
| batch 1 | 1.000 | 0.69 | −41% |

**Content recall** is the fraction of real content kept — the number that must
never slip. **Junk recall** is the fraction of boilerplate actually removed;
what it misses costs tokens, never correctness. Each batch was a fresh exam
nothing had been tuned on when first scored, then became training data — the
newest batch is always the honest one.

Add `--url-mode strip` for roughly 15 additional points of token cut with zero
text loss. With the `[model]` extra installed, a learned block-sequence scorer
raises junk recall further; it is capped so that it can never delete a block
on its own.

The benchmark harness, labeling pipeline, and mutation self-test live in
`bench/` — see `ARCHITECTURE.md`.
