Metadata-Version: 2.4
Name: contextractor
Version: 0.4.12
Summary: Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.
Project-URL: Homepage, https://www.contextractor.com
Project-URL: Documentation, https://www.contextractor.com/help/pypi/
Project-URL: Repository, https://github.com/contextractor/contextractor
Project-URL: Issues, https://github.com/contextractor/contextractor/issues
Project-URL: Changelog, https://github.com/contextractor/contextractor/releases
Project-URL: Apify Actor, https://apify.com/glueo/contextractor
Author-email: glueo <company@glueo.com>
License-Expression: Apache-2.0
Keywords: content-extraction,crawlee,crawler,markdown,playwright,trafilatura,web-scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: nodejs-wheel-binaries<25,>=24.16
Provides-Extra: test
Requires-Dist: pytest-asyncio>=1.4; extra == 'test'
Requires-Dist: pytest>=9.1; extra == 'test'
Requires-Dist: pyyaml>=6.0.3; extra == 'test'
Description-Content-Type: text/markdown

# contextractor

<img align="right" width="220" src="https://www.contextractor.com/media/cover-mini.svg" alt="Contextractor" />

<strong>Details &amp; badges:</strong>
<a href="https://pypi.org/project/contextractor/"><img src="https://img.shields.io/pypi/v/contextractor.svg" alt="PyPI version" /></a>
<a href="https://pypi.org/project/contextractor/"><img src="https://img.shields.io/pypi/dm/contextractor.svg" alt="PyPI downloads" /></a>
<a href="https://github.com/contextractor/contextractor/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/contextractor.svg" alt="license" /></a>

<strong>Also available as:</strong>
<strong><a href="https://www.contextractor.com/">Online playground</a></strong>
[<a href="https://www.contextractor.com/">playground</a>, <a href="https://www.contextractor.com/help/web/">help</a>] |
<strong><a href="https://apify.com/glueo/contextractor?fpr=glueo">Apify Actor</a></strong>
[<a href="https://apify.com/glueo/contextractor?fpr=glueo">actor</a>, <a href="https://www.contextractor.com/help/apify/">help</a>] |
<strong><a href="https://www.npmjs.com/package/contextractor">NPM package</a></strong>
[<a href="https://www.npmjs.com/package/contextractor">package</a>, <a href="https://www.contextractor.com/help/npm/">help</a>] |
<strong><a href="https://github.com/contextractor/contextractor">Source code on GitHub</a></strong>

<strong>Social:</strong>
<strong><a href="https://x.com/contextractor">X (Twitter)</a></strong>

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

> **⚠️ Alpha / experimental.** This Python wrapper for Contextractor is currently
> an **alpha**, experimental release — not fully tested or officially supported,
> though still maintained.

Crawl web pages and extract clean main-content text — `txt`, `markdown`, `json`,
`html`, or raw `original` HTML — from Python. Built on [`rs-trafilatura`](https://github.com/Murrough-Foley/rs-trafilatura)
(extraction) and [Crawlee](https://crawlee.dev/) with Playwright or a pure-HTTP
Cheerio crawler (crawling).

This package is a thin, typed wrapper that **drives the bundled Node engine**. A
self-contained Node runtime is installed automatically as a dependency
(`nodejs-wheel-binaries`), so no separate Node.js install is required.

**Homepage & docs:** [www.contextractor.com](https://www.contextractor.com) ·
[Python guide](https://www.contextractor.com/help/pypi/)

## Contents

- [Why Contextractor](#why-contextractor)
- [Install](#install)
- [Quick start](#quick-start)
- [Single-page extraction](#single-page-extraction)
- [Options](#options)
- [Links](#links)
- [Contributing](#contributing)
- [License](#license)

## Why Contextractor

<!-- @generated:start name="why-contextractor" -->

<!-- This block is auto-generated by @contextractor/gen-md-regions. Do not edit. -->

Contextractor ships the [Rust port of Trafilatura](https://github.com/Murrough-Foley/rs-trafilatura)
as a native (napi-rs) binding — no Python extraction runtime. On the Scrapinghub article set it
scores an **F1 of 0.966** (precision 0.942, recall 0.991) — ahead of go-trafilatura (0.960) and the
original Python Trafilatura (0.958); see the
[benchmark write-up](https://www.contextractor.com/trafilatura/) for the methodology.

It is free and open source (Apache-2.0), runs locally with **no API key and no per-page credits**,
and its Markdown output is typically **80–90% fewer tokens** than the raw HTML — cheap to feed to
an LLM.

|                   | Contextractor                                    | Firecrawl                      | Jina Reader                                | Crawl4AI                                          |
| ----------------- | ------------------------------------------------ | ------------------------------ | ------------------------------------------ | ------------------------------------------------- |
| Extraction engine | rs-trafilatura (heuristic + ML routing)          | LLM / heuristic                | ReaderLM neural model                      | LLM / heuristic                                   |
| Runtime           | Rust + Node (no Python engine)                   | hosted API / self-host         | hosted API                                 | Python                                            |
| Surfaces          | Apify Actor · npm CLI · npm library · PyPI       | API · SDKs · self-hosted · MCP | API                                        | Python library · crwl CLI · Docker REST API · MCP |
| Output formats    | txt · markdown · json · html · original          | markdown · html · etc.         | markdown · html · text · screenshot · etc. | markdown · etc.                                   |
| Crawling          | Crawlee + Playwright (adaptive / browser / HTTP) | built-in                       | none (single URL)                          | built-in                                          |

<!-- @generated:end name="why-contextractor" -->

## Install

```bash
pip install contextractor
python -m contextractor install   # one-time: download the Chromium browser
```

Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64;
glibc ≥ 2.28), and Windows (x64). Requires Python 3.12+.

## Quick start

Crawl one or more URLs and write extracted files plus a `manifest.json` index to
`output_dir`:

```python
import contextractor

summary = contextractor.extract(
    ["https://en.wikipedia.org/wiki/Web_scraping"],
    save=["markdown-kvs"],
    output_dir="./out",
)
print(summary.succeeded, "of", summary.total)
```

Each `save` token is `<format>-<kvs|dataset>` — the format name plus the
destination — so it reads `markdown-kvs`, not bare `markdown` (the bare names
like `markdown` and `json` are what `extract_one(formats=[...])` takes instead).

An async variant, `aextract()`, takes the same arguments. Partial failures do
not raise — they are reflected in `summary.failed`; validation and real crawl
failures raise `ContextractorError`, and a missing browser raises
`MissingBrowserError` pointing you at `python -m contextractor install`.

## Single-page extraction

`extract_one()` crawls exactly one URL (no link-following) and returns the
extracted content directly — nothing is written to disk:

```python
import contextractor

markdown = contextractor.extract_one("https://en.wikipedia.org/wiki/Web_scraping")  # -> str

contents = contextractor.extract_one(                                # -> dict[str, str]
    "https://en.wikipedia.org/wiki/Web_scraping",
    formats=["markdown", "json", "original"],
)
```

With one requested format the return value is a `str`; with several it is a
`dict[str, str]`. `aextract_one()` is the async variant.

## Options

Every crawl option is a typed keyword argument; the `ExtractOptions` /
`ExtractOneOptions` `TypedDict`s are the authoritative list of keywords and
accepted values, surfaced to editors and type checkers (e.g. `save`, `mode`,
`proxy`, `cookies`, `headers`, `max_crawl_depth`, `globs`). See also the
[Python documentation](https://www.contextractor.com/help/pypi/). Each Python
keyword maps one-to-one onto a flag of the npm CLI, documented at
[contextractor.com/help/npm](https://www.contextractor.com/help/npm/).

Only `http`, `https`, `socks4`, and `socks5` proxy URLs are accepted; credentials
are never echoed in errors or logs. Set `CONTEXTRACTOR_NODE_PATH` to use a host
Node binary instead of the bundled runtime.

## Links

- Product site and docs: [contextractor.com](https://www.contextractor.com) ([Python help page](https://www.contextractor.com/help/pypi/))
- Source: [github.com/contextractor/contextractor](https://github.com/contextractor/contextractor)
- Questions and bug reports: [open an issue](https://github.com/contextractor/contextractor/issues)

## Contributing

Issues and pull requests are welcome at the
[issue tracker](https://github.com/contextractor/contextractor/issues). The
extraction engine, npm CLI, and Apify Actor all live in the same
[source repository](https://github.com/contextractor/contextractor).

## License

[Apache-2.0](https://github.com/contextractor/contextractor/blob/main/LICENSE)
