Metadata-Version: 2.5
Name: markdownee
Version: 0.8.11
Summary: Native Python crawling and content extraction with Crawlee and Trafilatura Core.
Project-URL: Homepage, https://www.markdownee.com
Project-URL: Documentation, https://www.markdownee.com/help/pypi/
Project-URL: Repository, https://github.com/markdownee/markdownee
Project-URL: Issues, https://github.com/markdownee/markdownee/issues
Author-email: glueo <company@glueo.com>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: content-extraction,crawlee,crawler,markdown,trafilatura,web-scraping
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: crawlee[beautifulsoup,httpx,playwright]<2,>=1.10.4
Requires-Dist: lxml<7,>=6.1.3
Requires-Dist: markdownify<2,>=1.2.3
Requires-Dist: pydantic<3,>=2.13.5
Requires-Dist: pyvips[binary]<4,>=3.2
Requires-Dist: trafilaturacore==0.8.11
Provides-Extra: test
Requires-Dist: build>=1.6.1; extra == 'test'
Requires-Dist: hatchling>=1.32.4; extra == 'test'
Requires-Dist: pytest-asyncio>=1.4; extra == 'test'
Requires-Dist: pytest-cov>=7.1; extra == 'test'
Requires-Dist: pytest>=9.1.1; extra == 'test'
Requires-Dist: ruff>=0.16.10; extra == 'test'
Description-Content-Type: text/markdown

# Markdownee

<table>
  <tbody>
    <tr>
      <td>
        <img class="align-right" align="right" width="220" src="https://www.markdownee.com/media/logo-opaque.svg" alt="Markdownee" />
        <a href="https://pypi.org/project/markdownee/"><img src="https://img.shields.io/pypi/v/markdownee.svg" alt="PyPI version" /></a>
        <a href="https://pypi.org/project/markdownee/"><img src="https://img.shields.io/pypi/dm/markdownee.svg" alt="PyPI downloads" /></a>
        <a href="https://github.com/markdownee/markdownee/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/markdownee.svg" alt="license" /></a>
        <h3>Also available as:</h3>
        <strong><a href="https://www.markdownee.com/">Online playground</a></strong> | <strong><a href="https://www.npmjs.com/package/@markdownee/markdownee">npm package CLI &amp; lib</a></strong> | <strong><a href="https://github.com/markdownee/markdownee">Source code on GitHub</a></strong> | <strong><a href="https://apify.com/markdownee/crawler?fpr=glueo">Apify Actor</a></strong>
        <h3>Docs</h3>
        <strong><a href="https://www.markdownee.com/help/getting-started/">Getting started</a></strong> | <strong><a href="https://www.markdownee.com/help/pypi/">Python library help</a></strong>
        <h3>Social</h3>
        <strong><a href="https://github.com/markdownee/markdownee">Star us on GitHub</a></strong> | <strong><a href="https://github.com/markdownee">Follow us on GitHub</a></strong>
      </td>
    </tr>
  </tbody>
</table>

Markdownee is a web scraper that crawls websites and saves their content as **Markdown**, HTML or plain text for LLMs, retrieval pipelines, and research datasets.

Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.

- Boilerplate removal uses the native Python implementation of [Trafilatura Core](https://www.trafilaturacore.com/), our **open-source extraction engine** based on [Trafilatura](https://www.markdownee.com/trafilatura/).

  The _Core_ in its name means it is reduced to _one task_: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.

  Core's TypeScript engine ports the original Python [Trafilatura](https://github.com/adbar/trafilatura), with [go-trafilatura](https://github.com/markusmobius/go-trafilatura) as a DOM translation aid. Its native Python engine translates that port.

- [Crawlee](https://crawlee.dev/) handles crawling and uses [Playwright](https://playwright.dev/) for browser rendering.

- Two language versions — **TypeScript** and **Python**: self-host with the [npm CLI](https://www.markdownee.com/help/npm-cli/), [npm library](https://www.markdownee.com/help/npm-library/), or [Python library](https://www.markdownee.com/help/pypi/), or use the [hosted Apify Actor](https://apify.com/markdownee/crawler?fpr=glueo). Source code is on [GitHub](https://github.com/markdownee/markdownee).

- Save image files with the optional **image downloading** mode.

This package provides the native Python library, using Crawlee Python and Python Trafilatura Core. Python exposes library APIs only; see the [language differences](https://www.markdownee.com/help/pypi/) for its supported crawl controls.

## Install

Use Python 3.12 or newer:

```bash
python -m pip install markdownee
```

Tested PyPI platforms are macOS and Linux on x64/ARM64, and Windows x64.
Native Windows ARM64 installation and HTTP fetching are verified on Python 3.12
and 3.13. `impit` builds from source there and needs a Rust/C++ build toolchain.
32-bit Windows (x86) is unsupported.

HTTP crawling needs no browser download. For Chromium crawling, install the
Playwright browser:

```bash
python -m playwright install chromium
```

Firefox has its own setup in [Python Help](https://www.markdownee.com/help/pypi/).
The package installs its dependencies, including `pyvips[binary]`; platforms without
its binary support need a compatible libvips installation.

## Extract a page

Save as `extract.py`, then run `python extract.py`:

```python
from markdownee import fetch

contents = fetch(
    "https://docs.python.org/3/tutorial/introduction.html",
    crawler_type="http",
    formats=["markdown"],
)
print(contents["markdown"])
```

The program prints the extracted article. `fetch()` follows no links; unavailable
formats are omitted and request failure raises `MarkdowneeError`. Format selectors
are `txt`, `markdown`, `html`, `minified-html`, and `original`; compact HTML uses the
result key `minified_html`. Original is _captured input_, not cleaned output.

Use `afetch()` in async applications. `crawl()` and `acrawl()` write a fresh run
directory and return a `CrawlSummary` with output and manifest paths. Image saving
belongs to these file-producing calls; `fetch()` rejects it.

See [Python Help](https://www.markdownee.com/help/pypi/) for complete examples,
options, browser controls, limits, cancellation, and language differences.

## Why Markdownee

<!-- @generated:start name="why-markdownee" -->

<!-- This block is auto-generated by @markdownee/gen-md-regions. Do not edit. -->

Optional image downloading keeps local assets with extracted content.

[Trafilatura Core](https://www.trafilaturacore.com/) removes boilerplate. Markdownee's
TypeScript and Python versions each use the matching Core implementation.
Core's TypeScript engine ports original Python
[Trafilatura](https://github.com/adbar/trafilatura), with
[go-trafilatura](https://github.com/markusmobius/go-trafilatura) as a DOM translation
aid. Its native Python engine translates that port.
[Crawlee](https://crawlee.dev/) drives [Playwright](https://playwright.dev/)
for browser rendering.

Use the [npm CLI](https://www.markdownee.com/help/npm-cli/),
[npm library](https://www.markdownee.com/help/npm-library/),
[native Python library](https://www.markdownee.com/help/pypi/), or
[hosted Apify Actor](https://apify.com/markdownee/crawler?fpr=glueo).

<!-- @generated:end name="why-markdownee" -->

## Support

Report problems through the [issue tracker](https://github.com/markdownee/markdownee/issues).

## License

Licensed under [Apache-2.0](https://github.com/markdownee/markdownee/blob/main/LICENSE).
