Metadata-Version: 2.4
Name: scrapeMM
Version: 0.8.2
Summary: Multimodal scraper for social media, internet archives, and the open web.
Author-email: Mark Rothermel <mark.rothermel@tu-darmstadt.de>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/multimodal-ai-lab/scrapeMM
Project-URL: Issues, https://github.com/multimodal-ai-lab/scrapeMM
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp~=3.14.3
Requires-Dist: async_lru
Requires-Dist: atproto~=0.0.71
Requires-Dist: beautifulsoup4
Requires-Dist: backports.zstd~=1.7.0
Requires-Dist: brotli
Requires-Dist: cryptography
Requires-Dist: curl_cffi~=0.16.3
Requires-Dist: ezmm~=0.5.2
Requires-Dist: firecrawl-py~=4.32.0
Requires-Dist: markdownify~=1.2.3
Requires-Dist: pip-system-certs
Requires-Dist: platformdirs~=4.11.8
Requires-Dist: playwright~=1.62.0
Requires-Dist: playwright-stealth
Requires-Dist: seleniumbase~=4.54.0
Requires-Dist: prompt_toolkit
Requires-Dist: PyYAML~=6.0.1
Requires-Dist: requests~=2.34.2
Requires-Dist: telethon~=1.45.0
Requires-Dist: TikTokResearchApi~=1.0.4
Requires-Dist: tqdm~=4.70.0
Requires-Dist: tweepy~=4.17.0
Requires-Dist: yt-dlp[curl-cffi,default]
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Dynamic: license-file

# scrapeMM: Multimodal Web Retrieval
Simple web scraper to asynchronously retrieve webpages and access social media contents, fetching text along with media, i.e., images and videos.

This library aims to help developers and researchers to easily access multimodal data from the web and use it for LLM processing.

## Setup
* **If you want to download videos**: Then, the installation of [ffmpeg](https://ffmpeg.org/) is highly recommended.
In Conda, you can install it with `conda install -c conda-forge ffmpeg`. Platforms like YouTube and
Facebook serve video and audio as separate streams, and merging them needs ffmpeg. Without it,
videos are downloaded **without sound**.
* **Install Playwright dependencies** (used by multiple integrations) running `playwright install` (add `--force` if an already installed version needs an update).

## Configure
To set the API secrets, run
```python
from scrapemm import configure_secrets
configure_secrets()
```

To set the Firecrawl URL, run
```python
from scrapemm import update_config
update_config(firecrawl_url="your_url")
```

### Archive.today Access
Archive.today guards its snapshots with strong anti-bot protection. You can circumvent it by setting cookies of sessions where you manually solved a CAPTCHA. scrapeMM does **not** solve CAPTCHAs.

To establish a session, run
```bash
python scripts/configure_archive_today.py
```
This opens a snapshot of each mirror (archive.today, archive.is, archive.ph, …) in
scrapeMM's own browser, one after another for all 6 different archive.today domains. Pass the check in that window each time; the
resulting cookies are stored and re-used automatically. 

**Running headless (e.g. on a server)?** The browser has to stay on that machine — the
clearance you earn is bound to the browser *and* IP address that earned it — but its
window can be brought to your screen. While waiting, the script prints an SSH command and
a URL for exactly that: it tunnels Chrome's debugging port and opens the page in your
local browser through Chrome's own DevTools frontend, where your clicks reach the remote
page. Nothing needs to be installed on the server. Give yourself enough time to set the
tunnel up: `python scripts/configure_archive_today.py 900`.

Alternatively, paste a cookie export (`cookies.txt` or JSON, from any cookie extension)
directly:
```python
from scrapemm import override_secret
override_secret("archive_today_cookie")  # Paste the export, then press Alt+Enter
```
Your cookies replace the ones shipped with scrapeMM.

## Usage

```python
from scrapemm import retrieve
import asyncio

if __name__ == "__main__":
    url = "https://www.snopes.com/fact-check/gauze-originate-from-gaza/"
    result = asyncio.run(retrieve(url))
    if result.success:
        print(result.get())
    else:
        print(result.errors)
```

`retrieve()` returns a `ScrapingResponse`. Its `result.get()` returns the content in the requested
format; `result.content` gives access to every format that was produced along the way:

| Attribute | Type | Content |
|---|---|---|
| `result.content.html` | `str` | The raw HTML code of the page |
| `result.content.markdown` | `str` | The page text in Markdown, media referenced by hyperlink |
| `result.content.multimodal` | `MultimodalSequence` | The page text in Markdown with all media downloaded and embedded |

Use `output_format` to tell scrapeMM which format you need (default: `"multimodal"`):

```python
result = asyncio.run(retrieve(url, output_format="html"))
print(result.content.html)
```

The formats are produced one after another — HTML, then Markdown, then the `MultimodalSequence` —
and scraping stops as soon as the requested format is reached. So the earlier formats come along
for free (whenever the used retrieval method had access to them), while nothing beyond the
requested format is computed and media gets downloaded only for `"multimodal"`. `result.success`
tells you whether the requested format could be produced.
`scrapeMM` will ask you for the **API secrets** needed for the integrations. You may skip them if you don't need them.

You will also be prompted to choose a **password** that is used to secure the secrets in an encrypted file.

## Caching
Successful retrievals are cached in memory for 24 hours, so scraping the same URL again is
instantaneous. The cache is *not* persisted, i.e., it is empty again after the process ended.
`result.from_cache` tells you whether a response came from the cache.

To change the caching duration (in seconds) for the current process, run
```python
from scrapemm import set_cache_ttl
set_cache_ttl(60 * 60)  # cache for one hour
set_cache_ttl(0)  # disable caching
```
Use `update_config(cache_ttl=3600)` instead to persist the duration across processes, and
`clear_cache()` to empty the cache. To bypass the cache for a single call, pass
`retrieve(url, use_cache=False)`.

## CAPTCHAs and Blacklisted Domains
Every scraped page is checked for CAPTCHA challenges (Cloudflare, reCAPTCHA, hCaptcha, DataDome,
AWS WAF, PerimeterX, and others). If a challenge was served instead of the page content, the
response carries a `CaptchaEncounteredError` and the URL's domain is put on a blacklist that is
persisted to `blacklist.yaml` in scrapeMM's config directory. Retrieving any URL of a blacklisted
domain then fails right away with an `UnsupportedDomainError` telling why the domain was
blacklisted.

```python
from scrapemm import get_blacklisted_domains, unblacklist_domain, blacklist_domain

get_blacklisted_domains()  # -> {"example.com": "Method decodo encountered a Cloudflare challenge. ..."}
unblacklist_domain("example.com")  # Retrieve that domain again
blacklist_domain("example.com", "Paywalled")  # Exclude a domain manually
```

A domain gets blacklisted only if *all* retrieval methods failed, so a CAPTCHA on one method does
not exclude a domain that another method can still scrape. Blacklisting applies to the registrable
domain, i.e., including all of its subdomains.

Domains that are served by an integration (`perma.cc`, `archive.today`, `x.com`, ...) are never
blacklisted automatically: their CAPTCHA gates are transient, so blacklisting would disable the
respective integration for good.

## How it works
```
Input:                                  Output:
URL (string)   -->   retrieve()   -->   MultimodalSequence
```
The `MultimodalSequence` is a sequence of Markdown-formatted text and media provided by the [ezMM](https://github.com/multimodal-ai-lab/ezmm) library.

Web scraping is done with [Firecrawl](https://github.com/mendableai/firecrawl) and [Decodo](https://decodo.com/).

Media is collected from `<img>` and `<video>` tags, from CSS background images, and from
embedded players of the common video platforms (an `<iframe>` pointing at YouTube, Vimeo,
Dailymotion, …), which are downloaded with yt-dlp. `max_video_size` caps those downloads
just like it caps the ones of the platform integrations.

## Supported Platforms
### Social Media
- ✅ X/Twitter
- ✅ Telegram
- ✅ Bluesky
- ✅ TikTok
- ✅ YouTube
- ✅️ Instagram: works for most content
- ✅️ Facebook
- ✅ Threads: posts only (profiles TBD)
- ✅ Reddit: posts only

### Archiving Services
- ✅ Perma.cc
- ✅ Archive.today: Rarely ending up in TimeoutErrors
- ✅ MediaVault (mvau.lt)
- ✅ Internet Archive (web.archive.org)
- ✅ AwesomeScreenshot.com
- ✅ Ghostarchive (ghostarchive.org)
