Metadata-Version: 2.4
Name: mdscrape
Version: 0.1.0
Summary: Turn any webpage or sitemap into clean Markdown for LLMs, AI agents, and RAG pipelines.
Author: Sourav Agarwal
License-Expression: MIT
Project-URL: Homepage, https://github.com/SouravKAgarwal/web-scraper
Project-URL: Repository, https://github.com/SouravKAgarwal/web-scraper
Project-URL: Issues, https://github.com/SouravKAgarwal/web-scraper/issues
Keywords: scraper,markdown,mcp,model-context-protocol,llm,rag,fastapi,trafilatura
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4>=4.15.0
Requires-Dist: curl-cffi>=0.16.3
Requires-Dist: fastapi>=0.115.0
Requires-Dist: markdownify>=1.2.3
Requires-Dist: mcp[cli]>=2.3.0
Requires-Dist: requests>=2.34.2
Requires-Dist: sse-starlette>=2.0.0
Requires-Dist: trafilatura>=2.2.0
Requires-Dist: uvicorn>=0.30.0

# AI Web Scraper

A robust web scraper that extracts webpage content as clean **Markdown**, served via a FastAPI backend with a real-time streaming web UI.

## Features

- **MCP Server** — Exposes scraping tools via the [Model Context Protocol](https://modelcontextprotocol.io/) so AI agents (Claude Desktop, Cursor, Antigravity, etc.) can scrape web pages directly
- **Real-time Web UI** — Custom HTML/CSS/JS dashboard powered by FastAPI and Server-Sent Events (SSE) for live progress updates
- **Smart content extraction** — Trafilatura-first with BeautifulSoup pre-processing for maximum coverage
- **Anti-bot bypass** — Uses `curl_cffi` to impersonate real browsers (Chrome TLS fingerprint)
- **Retry with backoff** — Automatic retries with exponential backoff on failed requests
- **Markdown output** — Clean Markdown with title, source URL, and extracted content
- **Rich metadata** — Returns author, date, description, site name, language, word count, and Open Graph image alongside content
- **Batch scraping** — Scrape multiple URLs from a text file or the web UI
- **Sitemap support** — Extract URLs from XML sitemaps (including sitemap indexes) with recursive resolution
- **URL validation** — Validates all URLs before attempting to fetch
- **Markdown cleanup** — Fixes header levels, merges broken definition lists, and removes empty bullets from Trafilatura output

## Installation

Requires **Python 3.12+** and [uv](https://docs.astral.sh/uv/).

```bash
# Clone the repository
git clone https://github.com/SouravKAgarwal/web-scraper.git
cd web-scraper

# Install dependencies
uv sync
```

## Usage

### Web UI (Recommended)

Launch the FastAPI server to use the web interface:

```bash
uvicorn server:app --host 0.0.0.0 --port 8000 --reload
```

This starts the server at `http://localhost:8000`. Open it in your browser to:

- Enter one or more URLs (one per line) or a sitemap URL
- Set an optional limit on the number of pages to scrape
- Watch real-time progress as each page is scraped
- View rendered Markdown results with syntax highlighting

### MCP Server (for AI Agents)

Run MDScrape as an [MCP](https://modelcontextprotocol.io/) server so AI coding assistants can scrape web pages directly:

```bash
uv run mcp_server.py
```

**Configure in Claude Desktop / Cursor / Antigravity:**

```json
{
  "mcpServers": {
    "mdscrape": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/web-scraper", "mcp_server.py"]
    }
  }
}
```

**Available tools:**

| Tool | Description |
|---|---|
| `scrape_url` | Scrape a single page → Markdown + metadata |
| `scrape_urls` | Scrape multiple pages in batch |
| `scrape_sitemap` | Crawl a sitemap and scrape all pages |
| `extract_sitemap_urls` | List all URLs in a sitemap without scraping |

### CLI Usage

`mdscrape` is globally installed on your system. You can run it directly from any directory:

```bash
# Scrape a single URL
mdscrape https://example.com/article

# Scrape multiple URLs from a file
mdscrape -f urls.txt

# Scrape with limit
mdscrape -f urls.txt -n 10

# Directly crawl and scrape pages from a sitemap
mdscrape -s https://example.com/sitemap.xml -n 20

# Extract URLs from a sitemap without scraping (print to stdout)
mdscrape -s https://example.com/sitemap.xml --urls-only
```



## Project Structure

```
web-scraper/
├── cli.py               # Main CLI interface (URLs, files, sitemaps)
├── mcp_server.py        # MCP server exposing scraping tools for AI agents
├── server.py            # FastAPI server serving Web UI and /scrape API
├── helpers/             # Utility functions
│   ├── http.py          # HTTP fetch with retries, URL validation
│   └── markdown.py      # Markdown post-processing and cleanup
├── services/            # Core business logic
│   ├── scraper.py       # Page fetching, BS4 + Trafilatura extraction
│   └── sitemap.py       # XML sitemap parsing & recursive traversal
├── static/              # Web UI assets
│   ├── index.html       # UI layout
│   ├── script.js        # SSE stream listener & renderer
│   └── style.css        # UI styling
├── pyproject.toml       # Project metadata & dependencies
├── uv.lock              # Dependency lockfile
└── .python-version      # Python version (3.12)
```

## How It Works

```mermaid
flowchart LR
    A["URL Input"] --> B["curl_cffi Fetch"]
    B --> C["BS4 Pre-process"]
    C --> D{"Trafilatura\nExtract"}
    D -- "Success" --> E["Clean Markdown"]
    D -- "Fail" --> F["Error"]
    E --> G["Return JSON\nwith Metadata"]
```

1. **Fetch** — Downloads the page HTML using `curl_cffi` with Chrome browser impersonation to bypass anti-bot measures. Retries up to 3 times with exponential backoff.
2. **Pre-process** — Unwraps nested tags inside `<pre>` blocks with BeautifulSoup to prevent code block fragmentation in Trafilatura's output.
3. **Extract** — Passes the cleaned HTML to Trafilatura for intelligent content extraction as Markdown (with links and images). Also extracts metadata (title, author, date, description, etc.).
4. **Clean** — Post-processes the Markdown: downgrades header levels, merges broken definition lists, and removes empty bullets.
5. **Return** — Returns structured JSON containing the Markdown content and all extracted metadata.

## API Reference

### `POST /scrape`

Streams scrape results via Server-Sent Events (SSE).

**Request body:**

```json
{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "limit": 0
}
```

**Request Example:**

```py
import json

import requests

result = {}


def scrape_result(urls: list, limit: int = 0) -> dict:
    resp = requests.post(
        url="http://localhost:8000/scrape",
        json={"urls": urls, "limit": limit},
        stream=True,
    )

    for line in resp.iter_lines(decode_unicode=True):
        if line.startswith("data:"):
            event = json.loads(line[len("data:") :])
            if event["type"] == "done":
                del event["type"]
                result.update(event)

    return result


if __name__ == "__main__":
    urls =  ["https://example.com/page1", "https://example.com/page2"]
    limit = 2

    result = scrape_result(urls, limit)
    print(json.dumps(result, indent=4))


```

| Field     | Type        | Description                                                    |
| --------- | ----------- | -------------------------------------------------------------- |
| `urls`    | `list[str]` | URLs to scrape, or a single sitemap URL if `sitemap` is `true` |
| `sitemap` | `bool`      | Treat the first URL as a sitemap and extract URLs from it      |
| `limit`   | `int`       | Max URLs to scrape (`0` = unlimited)                           |

**SSE event types:**

| `type`     | Description                                                      |
| ---------- | ---------------------------------------------------------------- |
| `progress` | Sent before scraping each URL (`current`, `total`, `url`)        |
| `result`   | Scraped data for one URL (or error)                              |
| `info`     | Informational messages (e.g., sitemap extraction count)          |
| `done`     | Final summary (`successful`, `failed`, `total`, `response_time`) |
| `error`    | Fatal error (e.g., no valid URLs)                                |

## CLI Reference

### `main.py`

| Flag            | Description                         | Default   |
| --------------- | ----------------------------------- | --------- |
| `url`           | A single URL to scrape (positional) | —         |
| `-f`, `--file`  | Text file with URLs (one per line)  | —         |
| `-s`, `--sitemap`| Sitemap XML URL to crawl           | —         |
| `-n`, `--limit` | Max number of URLs to process       | Unlimited |
| `--urls-only`   | Print sitemap URLs without scraping | False     |

> If no URL, file, or sitemap is provided, prompts for manual input.

Output is printed as JSON to stdout (or URLs list with `--urls-only`).

---

## License

MIT
