Metadata-Version: 2.4
Name: scrapeMM
Version: 0.6.5
Summary: Multimodal scraper for social media, internet archives, and the open web.
Author-email: Mark Rothermel <mark.rothermel@tu-darmstadt.de>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/multimodal-ai-lab/scrapeMM
Project-URL: Issues, https://github.com/multimodal-ai-lab/scrapeMM
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp~=3.14.1
Requires-Dist: async_lru
Requires-Dist: atproto
Requires-Dist: beautifulsoup4
Requires-Dist: backports.zstd
Requires-Dist: brotli
Requires-Dist: cryptography
Requires-Dist: curl_cffi~=0.15.0
Requires-Dist: ezmm~=0.5.1
Requires-Dist: firecrawl-py~=4.32.0
Requires-Dist: markdownify~=1.2.3
Requires-Dist: oauthlib
Requires-Dist: pip-system-certs
Requires-Dist: platformdirs~=4.10.0
Requires-Dist: playwright
Requires-Dist: playwright-stealth
Requires-Dist: pyvirtualdisplay
Requires-Dist: prompt_toolkit
Requires-Dist: PyYAML~=6.0.1
Requires-Dist: requests~=2.34.2
Requires-Dist: telethon~=1.44.0
Requires-Dist: TikTokResearchApi~=1.0.4
Requires-Dist: tqdm~=4.68.4
Requires-Dist: tweepy~=4.17.0
Requires-Dist: yt-dlp[curl-cffi,default]
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Dynamic: license-file

# scrapeMM: Multimodal Web Retrieval
Simple web scraper to asynchronously retrieve webpages and access social media contents, fetching text along with media, i.e., images and videos.

This library aims to help developers and researchers to easily access multimodal data from the web and use it for LLM processing.

## Setup
* **If you want to download videos**: Then, the installation of [ffmpeg](https://ffmpeg.org/) is highly recommended.
In Conda, you can install it with `conda install -c conda-forge ffmpeg`.
* **If you want to scrape Perma.cc archive records or Facebook photos**, you'll need to install playwright with `pip install playwright` and running `playwright install`.

## Configure
To set the API secrets, run
```python
from scrapemm import configure_secrets
configure_secrets()
```

To set the Firecrawl URL, run
```python
from scrapemm import update_config
update_config(firecrawl_url="your_url")
```

## Usage
```python
from scrapemm import retrieve
import asyncio

if __name__ == "__main__":
    url = "https://www.snopes.com/fact-check/gauze-originate-from-gaza/"
    result = asyncio.run(retrieve(url))
    if result.errors:
        print(result.errors)
    else:
        print(result.content)
```
`scrapeMM` will ask you for the **API secrets** needed for the integrations. You may skip them if you don't need them.

You will also be prompted to choose a **password** that is used to secure the secrets in an encrypted file.

## How it works
```
Input:                                  Output:
URL (string)   -->   retrieve()   -->   MultimodalSequence
```
The `MultimodalSequence` is a sequence of Markdown-formatted text and media provided by the [ezMM](https://github.com/multimodal-ai-lab/ezmm) library.

Web scraping is done with [Firecrawl](https://github.com/mendableai/firecrawl) and [Decodo](https://decodo.com/).

## Supported Platforms
### Social Media
- ✅ X/Twitter
- ✅ Telegram
- ✅ Bluesky
- ✅ TikTok
- ✅ YouTube
- (✅️) Instagram: works for most content
- ✅️ Facebook
- ❌ Threads: TBD
- ❌ Reddit: TBD

### Archiving Services
- ✅ Perma.cc
- (✅) Archive.today: Sometimes ending up in TimeoutErrors, generally pretty slow
- ✅ MediaVault (mvau.lt)
- ❌ Wayback Machine, Internet Archive (web.archive.org)
- ❌ AwesomeScreenshot.com
- ❌ Ghost Archive (ghostarchive.org)
