Metadata-Version: 2.5
Name: sluicer
Version: 0.6.0
Summary: Turn a web page into structured data with no model in the loop.
Project-URL: Homepage, https://github.com/Gi0tto/sluicer
Project-URL: Documentation, https://gi0tto.github.io/sluicer/
Project-URL: Source, https://github.com/Gi0tto/sluicer
Project-URL: Issues, https://github.com/Gi0tto/sluicer/issues
Project-URL: Changelog, https://github.com/Gi0tto/sluicer/blob/main/CHANGELOG.md
Project-URL: Discussions, https://github.com/Gi0tto/sluicer/discussions
Project-URL: Funding, https://github.com/sponsors/Gi0tto
Author: Nichita Briculschi
License-Expression: MIT AND CC-BY-SA-3.0 AND Unicode-3.0
License-File: LICENSE
License-File: LICENSES/CC-BY-SA-3.0.txt
License-File: LICENSES/Unicode-3.0.txt
Keywords: deterministic,extraction,html,json-ld,microdata,opengraph,schema-org,scraping,structured-data,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.2
Requires-Dist: lxml>=5.3
Provides-Extra: api
Requires-Dist: curl-cffi>=0.15; extra == 'api'
Requires-Dist: mcp>=2.1; extra == 'api'
Requires-Dist: protego>=0.3; extra == 'api'
Requires-Dist: scrapling[fetchers]>=0.4.6; extra == 'api'
Requires-Dist: starlette>=1.2; extra == 'api'
Requires-Dist: trafilatura>=2.0; extra == 'api'
Requires-Dist: uvicorn>=0.31.1; extra == 'api'
Provides-Extra: fetch
Requires-Dist: curl-cffi>=0.15; extra == 'fetch'
Requires-Dist: protego>=0.3; extra == 'fetch'
Requires-Dist: scrapling[fetchers]>=0.4.6; extra == 'fetch'
Provides-Extra: markdown
Requires-Dist: trafilatura>=2.0; extra == 'markdown'
Provides-Extra: mcp
Requires-Dist: curl-cffi>=0.15; extra == 'mcp'
Requires-Dist: mcp>=2.1; extra == 'mcp'
Requires-Dist: protego>=0.3; extra == 'mcp'
Requires-Dist: scrapling[fetchers]>=0.4.6; extra == 'mcp'
Requires-Dist: trafilatura>=2.0; extra == 'mcp'
Provides-Extra: microformats
Requires-Dist: mf2py>=2.0; extra == 'microformats'
Description-Content-Type: text/markdown

<!-- mcp-name: io.github.Gi0tto/sluicer -->

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/logo-dark.png">
    <img src="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/logo.png" alt="Sluicer" width="440">
  </picture>
</p>

<p align="center">
  <strong>Turn a web page into structured data. No model, no API key, no bill.</strong>
</p>

<p align="center">
  <a href="https://pypi.org/project/sluicer/"><img src="https://img.shields.io/pypi/v/sluicer" alt="PyPI"></a>
  <a href="https://pypi.org/project/sluicer/"><img src="https://img.shields.io/pypi/pyversions/sluicer" alt="Python versions"></a>
  <a href="https://github.com/Gi0tto/sluicer/actions/workflows/ci.yml"><img src="https://github.com/Gi0tto/sluicer/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://github.com/Gi0tto/sluicer/blob/main/LICENSE"><img src="https://img.shields.io/badge/licence-MIT%2C%20two%20data%20files%20their%20own-yellow.svg" alt="MIT, two data files under their own licences"></a>
  <img src="https://img.shields.io/badge/LLM%20calls-none-blue" alt="no LLM calls">
</p>

<p align="center">
  <a href="https://gi0tto.github.io/sluicer/"><b>Documentation</b></a> ·
  <a href="https://github.com/Gi0tto/sluicer/blob/main/docs/agents.md"><b>In your agent</b></a> ·
  <a href="https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard.md"><b>Scoreboards</b></a> ·
  <a href="https://github.com/Gi0tto/sluicer/blob/main/docs/why.md"><b>Why Sluicer</b></a> ·
  <a href="https://github.com/Gi0tto/sluicer/discussions"><b>Discussions</b></a> ·
  <a href="https://github.com/Gi0tto/sluicer/blob/main/CHANGELOG.md"><b>Changelog</b></a>
</p>

---

Sluicer reads the structured data a web page already declares -- JSON-LD,
microdata, RDFa, OpenGraph and more -- and merges it into one record per thing,
every value naming the vocabulary and the place on the page it came from. No
model reads the page, so the same page always gives the same answer.

<p align="center">
  <img src="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/inspect.svg" alt="sluicer inspect on a product page: one record merged from JSON-LD, microdata and OpenGraph; a summary in which every answer names its source; and the page's two prices, 41.90 and 39.90, reported as a conflict" width="860">
</p>

## Highlights

- **Eight vocabularies, one record.** JSON-LD, microdata, RDFa, Dublin Core,
  OpenGraph, the Twitter card, HTML's meta names and microformats2, merged
  into one record per thing.
- **Provenance for every value.** The vocabulary, the key, and the place on the
  page: an XPath, and inside JSON-LD a pointer to the very value.
- **A summary of 25 questions** -- title, author, date, price, currency,
  availability, GTIN and the rest -- each answer naming where it came from, and
  a **conflict** reported when the page answers one of them two ways.
- **Extractors that fail loudly.** Learn one from a few pages of a template; a
  page that drifted exits 3 instead of returning nulls for weeks, and `heal`
  says what moved where.
- **Deterministic and light.** No model and no key: the 511 pages of the WCXB
  test set are read in 1.4 s, and the base install is three packages.
- **Polite by construction.** It announces itself, obeys robots.txt and
  `Crawl-delay`, waits a site's `Retry-After` in a crawl, and honours TDMRep
  reservations when asked.
- **Made for agents.** An MCP server with ten read-only tools, tried in Claude
  Code, Codex and Gemini CLI, and the same tools over HTTP for any language.
- **Measured in public, losses included.** Five scoreboards against
  trafilatura, newspaper4k, metascraper, extruct, Zyte and Diffbot.

## Install

```bash
uv pip install "sluicer[fetch,markdown,mcp]"
```

With pip, `pip install "sluicer[fetch,markdown,mcp]"`; as a command in an
environment of its own, `uv tool install "sluicer[fetch,markdown,mcp]"`. The
base install, `uv pip install sluicer`, reads HTML you already have with `lxml`
and `click` alone. Each extra adds one job:

| extra | adds |
|---|---|
| `fetch` | fetching: plain HTTP first, a browser only when a measurement says the page needs one |
| `markdown` | a page's main content as markdown, by trafilatura |
| `mcp` | the MCP server, with `fetch` and `markdown` |
| `api` | the HTTP API, with `mcp` |
| `microformats` | microformats2, which is off by default |

For the browser rung, once: `uvx --from "sluicer[fetch]" scrapling install`.
Without it, plain HTTP still works, and a page that wanted a browser comes back
from the HTTP rung with the failed climb recorded.

## Quick start

The product page in the picture above is
[`examples/brake-pads.html`](https://github.com/Gi0tto/sluicer/blob/main/examples/brake-pads.html).

```python
>>> import sluicer
>>> page = open("examples/brake-pads.html", "rb").read()
>>> result = sluicer.extract(page, url="https://example.com/p/bp-2210")
>>> price = result.summary["price"]
>>> price.value, price.source, price.key
('41.90', 'jsonld', 'Product.offers.price')
>>> price.where
'/html/head/script[1]#/offers/price'
>>> result.normalised
{'price': '41.90', 'currency': 'EUR', 'gtin': '4006381333931'}
>>> [(answer.value, answer.source) for answer in result.conflicts[0].answers]
[('41.90', 'jsonld'), ('39.90', 'opengraph')]
```

The page describes one product in three vocabularies; `result.records` holds it
once, each field with its source and place. The same reading from the command
line is `sluicer extract` for JSON, or `sluicer inspect` for the picture above.

## See it meet a redesign

<p align="center">
  <img src="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/demo.gif" alt="An extractor learnt from a software directory in January 2016 replays a page of February 2016 and exits 0; on the page of June 2024, after the site's redesign, it fails loudly with exit 3, and heal says where the listing and each field went, with how many learnt values it found there" width="860">
</p>

A real site's software directory, as the Wayback Machine kept it. An extractor
learnt from two pages of January 2016 replays a page of February 2016 and exits
0; on the page of June 2024, after the site's redesign, it fails loudly and
exits 3, where a selector would have returned nulls. `heal` then says where the
listing and each field went, and on how many of the values it learnt each move
rests. Every command ran for real; `scripts/demo.py` records them again.

## Use it

### From the command line

```bash
sluicer extract page.html                           # a file, a URL, or - for stdin
sluicer inspect https://example.com/product         # the same, for a person to read
sluicer extract listing.html --induce               # rows of a page that declares nothing
sluicer markdown https://example.com/article        # the readable content
sluicer diff yesterday.html https://shop.example/p  # what changed, and where from
sluicer diff URL URL --at 2024-01                   # since the Wayback Machine's capture
sluicer extract URL --cache ~/.cache/sluicer        # ask the site if it changed (304)
sluicer audit https://example.com/product           # its markup against Google's documentation

sluicer compile page1.html page2.html -o shop.json  # learn an extractor
sluicer compile p1.html p2.html -o shop.json --want price=41.90 --want title="Brake pads"
sluicer run shop.json https://shop.example/c?p=7    # replay it, checked
sluicer heal shop.json https://shop.example/c -o shop.json  # after a redesign

sluicer map https://shop.example/                   # a site's addresses, from its sitemaps
sluicer crawl https://shop.example/ -o shop.jsonl   # follow its links, politely; --resume
sluicer batch urls.txt -o pages.jsonl               # read a list, one JSON line per page
sluicer warc crawl.warc.gz > pages.jsonl            # the pages a web archive holds
sluicer feed https://blog.example/                  # a feed's items, from the page that declares it
```

Exit codes follow grep: 0 found, 1 nothing declared, 2 could not read, and 3
for a page that broke its extractor, a heal that lost a field, or an audit that
found a documented rule broken. A drifted page never exits 0.

### In your agent

```bash
claude mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp   # Claude Code
codex mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp    # Codex
```

Cursor, VS Code, Gemini CLI, Claude Desktop and Zed, and LangChain, the OpenAI
Agents SDK and Pydantic AI, are in
[In your agent](https://github.com/Gi0tto/sluicer/blob/main/docs/agents.md).
The repository is also a Claude Code plugin, with a skill in the open Agent
Skills format that Codex reads too. The server has ten tools:

| tool | answers |
|---|---|
| `extract_declared` | what a page declares, with provenance, the summary and its conflicts |
| `page_markdown` | the page's main content as markdown |
| `fetch_page` | the page's HTML, and whether it took plain HTTP or a browser |
| `compile_extractor` | an extractor learnt from pages of one template |
| `run_extractor` | an extractor replayed on a page, checked against what it learnt |
| `heal_extractor` | the extractor learnt again after a redesign, and what moved |
| `audit_page` | the page's markup against what Google documents |
| `read_feed` | a feed's items: RSS, Atom or JSON Feed |
| `map_site` | a site's addresses, from its sitemaps or its start page's links |
| `crawl_site` | a site's pages, following its links, each summarised |

Every answer carries `ok`, true exactly when it can be used as it is, and an
output schema; every tool says in its annotations that it only reads. The
server fetches nothing on `localhost`, a private network or a cloud's metadata
endpoint -- redirects and a browser's requests included -- unless started with
`SLUICER_ALLOW_PRIVATE=1`.

### From any other language

`sluicer serve` answers the same tools over HTTP, `POST /v1/tools/<name>` with
the tool's arguments as JSON, and describes them at `/openapi.json`. It listens
on loopback; anywhere else it needs `SLUICER_API_TOKEN`. See
[the HTTP API](https://github.com/Gi0tto/sluicer/blob/main/docs/http-api.md).

```bash
sluicer serve                                       # 127.0.0.1:8000
curl -s http://127.0.0.1:8000/v1/tools/extract_declared \
  -H 'Content-Type: application/json' -d '{"html_or_url": "https://example.com/p"}'
```

## How it compares

| | Sluicer | extruct | trafilatura, newspaper4k | CSS-selector scrapers | LLM extraction |
|---|---|---|---|---|---|
| Structured data merged into one record per thing | yes | no, one list per vocabulary | for a few metadata fields | no | depends on the prompt |
| Where each value came from | vocabulary, key and place | no | no | no | no |
| Bylines and dates from the visible text | no, by design | no | yes | where you write selectors | yes |
| The same answer for the same page | yes | yes | yes | yes | not guaranteed |
| A site's changed layout noticed | fails loudly, then `heal` | -- | -- | not by itself | not by itself |
| A model or an API key needed | no | no | no | no | yes |

`from sluicer.compat import extruct` answers extruct's own calls in its own
shapes, for code written against it: see
[moving from extruct](https://github.com/Gi0tto/sluicer/blob/main/docs/extruct.md).
[Why Sluicer](https://github.com/Gi0tto/sluicer/blob/main/docs/why.md) has the
full comparison, and says when another tool is the better choice.

## Measured, losses included

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/dates-dark.svg">
    <img src="https://raw.githubusercontent.com/Gi0tto/sluicer/main/docs/assets/dates-light.svg" alt="Publication dates on 360 pages as served: Sluicer finds 0.780 and is right on 0.734 of its answers, with 36 dates invented; trafilatura finds 0.855 and is right on 0.393, with 187 invented; metascraper finds 0.384 and is right on 0.271, with 80; newspaper4k finds 0.786 and is right on 0.658, with 54" width="760">
  </picture>
</p>

Sluicer reads only what a page declares, so it answers less often than tools
that also read the visible page, and is wrong less often when it answers. The
summary beside them on the 511 annotated test pages of the public WCXB corpus,
where hit rate is right answers over the pages that carry a label and an
invention is an answer on a page whose label is empty:

| | title | author | date | dates invented | seconds | packages |
|---|---|---|---|---|---|---|
| **sluicer 0.6.0** | 0.727 | 0.532 | 0.581 | **8** | **1.4** | **3** |
| trafilatura 2.2.0 | 0.745 | 0.750 | 0.838 | 216 | 16.2 | 17 |
| newspaper4k 0.9.6 | 0.768 | 0.532 | 0.645 | 52 | 29.6 | 22 |
| metascraper 5.58.1 | 0.654 | 0.787 | 0.374 | 84 | 2.5 | 125 |

WCXB strips every `<script>`, and with it JSON-LD, the vocabulary Sluicer reads
first. The same labels on the 360 of those pages a web archive holds as their
servers sent them, scripts intact:

| as served | title | author | date | right when it answers a date | dates invented |
|---|---|---|---|---|---|
| **sluicer 0.6.0** | 0.708 | 0.690 | 0.780 | **0.734** | **36** |
| trafilatura 2.2.0 | 0.756 | 0.860 | 0.855 | 0.393 | 187 |
| newspaper4k 0.9.6 | 0.767 | 0.705 | 0.786 | 0.658 | 54 |
| metascraper 5.58.1 | 0.667 | 0.845 | 0.384 | 0.271 | 80 |

33 of Sluicer's 36 invented dates are dates the page declares in its own
JSON-LD and does not show a reader, which is what the labels describe. The
other scoreboards, each with its method and the commands that regenerate it:

- **[Products](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-products.md)**,
  Zyte's benchmark of 140 pages scored by Zyte's evaluator: price F1 0.750 and
  availability F1 0.907, against 0.685 and 0.626 for the extruct baseline Zyte
  published, and 0.918 and 0.957 for Zyte's paid API, which reads the visible
  page with trained models.
- **[News in many languages](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-news.md)**,
  263 pages from 42 countries' publishers in 21 languages: the most titles
  right of the four tools, 0.871, and never a wrong date when it answers one.
  trafilatura finds more authors, 0.879 against 0.829, and 12 of the 17 it
  finds and Sluicer does not are the paper's own name, which Sluicer does not
  count an author.
- **[trafilatura's evaluation set](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-evaldata.md)**,
  990 pages it annotated to measure itself: the most titles right again, 0.776
  against 0.738, and fewer of the bylines and dates the annotators read off the
  visible page, 0.468 and 0.584 against 0.669 and 0.865.
- **[Drift](https://github.com/Gi0tto/sluicer/blob/main/docs/drift.md)**:
  extractors learnt on Wayback Machine captures of 25 sites and replayed on
  later ones, 44 pairs; none failed silently and none raised a false alarm.

## FAQ

<details>
<summary><b>Does Sluicer use an LLM anywhere?</b></summary>

No. A test fails the build if a model client is ever imported, and no feature
needs anybody's API key. That is what makes the same page give the same answer,
and a run cost only CPU.
</details>

<details>
<summary><b>What if a page declares nothing?</b></summary>

`sluicer extract --induce` reads the rows the page's markup repeats, a listing's
cards or a table's lines, marked `source="induced"`. And
`sluicer compile --want price=41.90` learns where values sit from examples of
them, on listings and on product pages that declare nothing; what it learns is
checked on every page it reads, so a price slot that starts saying "Add to
basket" fails instead of being returned.
</details>

<details>
<summary><b>Will it get past a site's bot protection?</b></summary>

It is not built to. Every request says `Sluicer/<version>` with the project's
address, robots.txt is obeyed, and a crawl never climbs to the one rung that
does not announce itself, which a single-page command reaches only with
`--stealth`. In a crawl, a site that answers 429 or 503 is asked again only
after its `Retry-After`.
</details>

<details>
<summary><b>Does it work in my language?</b></summary>

Declared data is the same in every language, and the news scoreboard measures
21 of them; no miss there was down to a page's language. Dates written in words
are read with the month names of the 430 languages and regions the Unicode CLDR
covers at its modern level, and the numbers with units Chinese, Japanese and
Korean write.
</details>

<details>
<summary><b>Can I get a guess from the visible page when nothing is declared?</b></summary>

Not in the summary: there such an answer would look exactly like a declared one.
[`examples/04_a_guess_from_the_visible_page.py`](https://github.com/Gi0tto/sluicer/blob/main/examples/04_a_guess_from_the_visible_page.py)
puts trafilatura's guess beside what the page declares, named a guess, and the
known limits say how often it is right.
</details>

<details>
<summary><b>Is it ready for production?</b></summary>

It is Beta: the interface may still change before 1.0, and every change is in
the [changelog](https://github.com/Gi0tto/sluicer/blob/main/CHANGELOG.md). Each
release passes the full test suite on Python 3.10 to 3.14 and property tests
that draw thousands of hostile pages, and is measured again on every
scoreboard, before it is tagged.
[Known limits](https://github.com/Gi0tto/sluicer/blob/main/docs/known-limits.md)
lists what it does not do, measured.
</details>

## Principles

- **No LLM call, anywhere in the path.** A test fails the build if a model
  client is ever imported.
- **No paid API.** A feature that needs somebody's key does not ship.
- **Deterministic.** The same page always gives the same answer, which is what
  makes the scoreboards reproducible.
- **Honest about failure.** A page that cannot be read says so, and nothing
  returns a plausible answer where the truth was unavailable.

## Documentation

At <https://gi0tto.github.io/sluicer/>, or in the repository:
[Why Sluicer](https://github.com/Gi0tto/sluicer/blob/main/docs/why.md) ·
[Extractors](https://github.com/Gi0tto/sluicer/blob/main/docs/extractors.md) ·
[In your agent](https://github.com/Gi0tto/sluicer/blob/main/docs/agents.md) ·
[HTTP API](https://github.com/Gi0tto/sluicer/blob/main/docs/http-api.md) ·
[Audit](https://github.com/Gi0tto/sluicer/blob/main/docs/audit.md) ·
[Crawling](https://github.com/Gi0tto/sluicer/blob/main/docs/crawling.md) ·
[Scoreboard](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard.md) ·
[Scoreboard, as served](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-served.md) ·
[Scoreboard, products](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-products.md) ·
[Scoreboard, news](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-news.md) ·
[Scoreboard, trafilatura's set](https://github.com/Gi0tto/sluicer/blob/main/docs/scoreboard-evaldata.md) ·
[Drift](https://github.com/Gi0tto/sluicer/blob/main/docs/drift.md) ·
[Moving from extruct](https://github.com/Gi0tto/sluicer/blob/main/docs/extruct.md) ·
[Known limits](https://github.com/Gi0tto/sluicer/blob/main/docs/known-limits.md) ·
[Design notes](https://github.com/Gi0tto/sluicer/blob/main/docs/design-notes.md) ·
[Examples](https://github.com/Gi0tto/sluicer/tree/main/examples) ·
[Roadmap](https://github.com/Gi0tto/sluicer/blob/main/ROADMAP.md) ·
[Changelog](https://github.com/Gi0tto/sluicer/blob/main/CHANGELOG.md) ·
[Security](https://github.com/Gi0tto/sluicer/blob/main/SECURITY.md) ·
[Contributing](https://github.com/Gi0tto/sluicer/blob/main/CONTRIBUTING.md)

## Community

Questions, ideas and what you built with Sluicer go to
[Discussions](https://github.com/Gi0tto/sluicer/discussions). A page Sluicer
read wrong is [an issue](https://github.com/Gi0tto/sluicer/issues/new/choose),
with the page attached, so that the fix comes with a test. A vulnerability is
reported privately, as
[SECURITY.md](https://github.com/Gi0tto/sluicer/blob/main/SECURITY.md) says.
[CONTRIBUTING.md](https://github.com/Gi0tto/sluicer/blob/main/CONTRIBUTING.md)
says how to set up, what CI checks and where each part of the code lives.

Sluicer is built and kept up by one person. If it saves you time or a bill,
[sponsoring it](https://github.com/sponsors/Gi0tto) keeps the scoreboards
measured and the extractors honest as the web changes.

## Licence

MIT, with no vendored code, and two exceptions: `sluicer/audit/schema_org.py`
holds schema.org's type and enumeration names, which schema.org publishes
under CC BY-SA 3.0, and `sluicer/calendar_names.py` holds CLDR's month and
weekday names, which Unicode publishes under the Unicode License v3; each of
the two files is distributed under its own (the package's licence expression
is `MIT AND CC-BY-SA-3.0 AND Unicode-3.0`). The base install needs `lxml`
and `click`, both BSD-3-Clause. The extras pull a wider tree that is not all
permissive: `tld` is tri-licensed MPL-1.1, GPL-2.0-only or LGPL-2.1-or-later,
`orjson` is MPL-2.0 alongside Apache-2.0 or MIT, and `certifi` is MPL-2.0. CI
lists every licence in that tree and fails on one nobody has read; see
[the licence notes](https://github.com/Gi0tto/sluicer/blob/main/docs/known-limits.md).
