Metadata-Version: 2.5
Name: markdownee
Version: 0.8.6
Summary: Native Python crawling and content extraction with Crawlee and Trafilatura Core.
Project-URL: Homepage, https://www.markdownee.com
Project-URL: Documentation, https://www.markdownee.com/help/pypi/
Project-URL: Repository, https://github.com/markdownee/markdownee
Project-URL: Issues, https://github.com/markdownee/markdownee/issues
Author-email: glueo <company@glueo.com>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: content-extraction,crawlee,crawler,markdown,trafilatura,web-scraping
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: crawlee[beautifulsoup,httpx,playwright]<2,>=1.10
Requires-Dist: lxml<7,>=6
Requires-Dist: markdownify<2,>=1.2
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: pyvips[binary]<4,>=3.2
Requires-Dist: trafilaturacore==0.8.6
Provides-Extra: test
Requires-Dist: build; extra == 'test'
Requires-Dist: hatchling; extra == 'test'
Requires-Dist: pytest-asyncio>=1; extra == 'test'
Requires-Dist: pytest-cov>=7; extra == 'test'
Requires-Dist: pytest>=9; extra == 'test'
Requires-Dist: ruff>=0.16; extra == 'test'
Description-Content-Type: text/markdown

# Markdownee

<table>
  <tbody>
    <tr>
      <td>
        <img class="align-right" align="right" width="220" src="https://www.markdownee.com/media/logo.svg" alt="Markdownee" />
        <a href="https://pypi.org/project/markdownee/"><img src="https://img.shields.io/pypi/v/markdownee.svg" alt="PyPI version" /></a>
        <a href="https://pypi.org/project/markdownee/"><img src="https://img.shields.io/pypi/dm/markdownee.svg" alt="PyPI downloads" /></a>
        <a href="https://github.com/markdownee/markdownee/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/markdownee.svg" alt="license" /></a>
        <h3>Also available as:</h3>
        <strong><a href="https://www.markdownee.com/">Online playground</a></strong> | <strong><a href="https://apify.com/markdownee/crawler?fpr=glueo">Apify Actor</a></strong> | <strong><a href="https://www.npmjs.com/package/@markdownee/markdownee">npm package CLI &amp; lib</a></strong> | <strong><a href="https://github.com/markdownee/markdownee">Source code on GitHub</a></strong>
        <h3>Docs</h3>
        <strong><a href="https://www.markdownee.com/help/getting-started/">Getting started</a></strong> | <strong><a href="https://www.markdownee.com/help/pypi/">Python library help</a></strong>
        <h3>Social</h3>
        <strong><a href="https://github.com/markdownee/markdownee">Star us on GitHub</a></strong> | <strong><a href="https://github.com/markdownee">Follow us on GitHub</a></strong>
      </td>
    </tr>
  </tbody>
</table>

Markdownee is a web scraper that crawls websites and saves their content as **Markdown**, HTML or plain text for LLMs, retrieval pipelines, and research datasets.

Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.

- Boilerplate removal is powered by [Trafilatura Core](https://www.trafilaturacore.com/), our **open-source fork** of [Trafilatura](https://www.markdownee.com/trafilatura/).

  The _Core_ in its name means it is reduced to _one task_: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.

  Trafilatura Core is ported from the original Python [Trafilatura](https://github.com/adbar/trafilatura), with [go-trafilatura](https://github.com/markusmobius/go-trafilatura) as a DOM translation aid.

- [Crawlee](https://crawlee.dev/) handles crawling and uses [Playwright](https://playwright.dev/) for browser rendering.

- Two language versions — **TypeScript** and **Python**: self-host with the [npm CLI](https://www.markdownee.com/help/npm-cli/), [npm library](https://www.markdownee.com/help/npm-library/), or [Python library](https://www.markdownee.com/help/pypi/), or use the [hosted Apify Actor](https://apify.com/markdownee/crawler?fpr=glueo). Source code is on [GitHub](https://github.com/markdownee/markdownee).

- Save image files with the optional **image downloading** mode.

This package provides the native Python library, using Crawlee Python and Python Trafilatura Core. Python exposes library APIs only; see the [language differences](https://www.markdownee.com/help/pypi/) for its supported crawl controls.

## Install

Use Python 3.12 or newer:

```bash
python -m pip install markdownee
```

Tested PyPI platforms are macOS and Linux on x64/ARM64, and Windows x64.
Native Windows ARM64 support is planned but currently blocked by `impit` packaging;
32-bit Windows (x86) is unsupported.

HTTP crawling needs no browser download. For Chromium crawling, install the
Playwright browser:

```bash
python -m playwright install chromium
```

Firefox has its own setup in [Python Help](https://www.markdownee.com/help/pypi/).
The package installs its dependencies, including `pyvips[binary]`; platforms without
its binary support need a compatible libvips installation.

## Extract a page

Save as `extract.py`, then run `python extract.py`:

```python
from markdownee import fetch

contents = fetch(
    "https://docs.python.org/3/tutorial/introduction.html",
    crawler_type="http",
    formats=["markdown"],
)
print(contents["markdown"])
```

The program prints the extracted article. `fetch()` follows no links; unavailable
formats are omitted and request failure raises `MarkdowneeError`. Format selectors
are `txt`, `markdown`, `html`, `minified-html`, and `original`; compact HTML uses the
result key `minified_html`. Original is _captured input_, not cleaned output.

Use `afetch()` in async applications. `crawl()` and `acrawl()` write a fresh run
directory and return a `CrawlSummary` with output and manifest paths. Image saving
belongs to these file-producing calls; `fetch()` rejects it.

See [Python Help](https://www.markdownee.com/help/pypi/) for complete examples,
options, browser controls, limits, cancellation, and language differences.

## Why Markdownee

<!-- @generated:start name="why-markdownee" -->

<!-- This block is auto-generated by @markdownee/gen-md-regions. Do not edit. -->

Choose what to fetch, _which content to retain_, and where to save it. Page limits,
link filters, and separate controls for images, tables, links, and user comments
keep those choices explicit. Optional image downloading keeps local assets with
extracted content.

[Trafilatura Core](https://www.trafilaturacore.com/) ports original Python
[Trafilatura](https://github.com/adbar/trafilatura), with
[go-trafilatura](https://github.com/markusmobius/go-trafilatura) as a DOM translation
aid. [Crawlee](https://crawlee.dev/) drives [Playwright](https://playwright.dev/)
for browser rendering.

Use the [npm CLI](https://www.markdownee.com/help/npm-cli/),
[npm library](https://www.markdownee.com/help/npm-library/),
[native Python library](https://www.markdownee.com/help/pypi/), or
[hosted Apify Actor](https://apify.com/markdownee/crawler?fpr=glueo).

<!-- @generated:end name="why-markdownee" -->

## Support

Report problems through the [issue tracker](https://github.com/markdownee/markdownee/issues).

## License

Licensed under [Apache-2.0](https://github.com/markdownee/markdownee/blob/main/LICENSE).
