Metadata-Version: 2.4
Name: scraping-mcp
Version: 0.1.0
Summary: MCP server for Scraping — REST API wrapper for LLM agents.
Project-URL: Homepage, https://github.com/Nodirbek2001/scraper
Project-URL: Issues, https://github.com/Nodirbek2001/scraper/issues
Author-email: Scraping Team <support@scraping.io>
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27.0
Requires-Dist: mcp<2.0.0,>=1.0.0
Requires-Dist: pydantic-settings>=2.1.0
Requires-Dist: pydantic>=2.5.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: respx>=0.20; extra == 'dev'
Description-Content-Type: text/markdown

# scraping-mcp

MCP (Model Context Protocol) server for [Scraping](https://scraping.io) — lets LLM agents (Cursor, Claude Desktop, Claude Code, Zed) call our scraping tools directly.

## Install

```bash
pip install scraping-mcp
# or with pipx for isolation:
pipx install scraping-mcp
```

Requires Python 3.11+.

## Setup

Get an API key at [dashboard/keys](https://api.scraping.io/dashboard/keys).

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) / `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": {
        "SCRAPING_API_KEY": "fc_live_abc123...",
        "SCRAPING_BASE_URL": "https://api.scraping.io"
      }
    }
  }
}
```

Restart Claude Desktop. Tools will appear as "scraping".

### Cursor

Create `.cursor/mcp.json` in the repo root:

```json
{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}
```

### Claude Code

Add to `~/.claude.json` (user scope):

```json
{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}
```

## Tools

| Tool | Description |
|---|---|
| `scrape(url, format?, cookies?, timeout_s?)` | Scrape single URL. Returns markdown or `{status:pending, task_id}` on timeout. |
| `crawl(urls[], cookies?, timeout_s?)` | Batch-scrape up to 500 URLs. Returns items or `{status:pending, crawl_id}` on timeout. |
| `get_scrape_status(task_id)` | Check status of a scrape task. |
| `get_crawl_status(crawl_id, include_items?)` | Check status + items of a crawl batch. |
| `get_balance()` | Current credit balance. |

### Examples

```
Agent: "Scrape the README at https://github.com/hashicorp/terraform"
-> calls scrape(url="https://github.com/hashicorp/terraform")
-> returns {status: "finished", result: {markdown: "# Terraform..."}}

Agent: "Scrape these 50 docs pages and summarize"
-> calls crawl(urls=[...50 urls...])
-> returns {status: "finished", total: 50, items: [...]}

Agent: "Check my credit balance"
-> calls get_balance()
-> returns {balance: 87}
```

## Environment variables

| Name | Required | Default | Description |
|---|---|---|---|
| `SCRAPING_API_KEY` | yes | — | Your API key (starts with `fc_live_`) |
| `SCRAPING_BASE_URL` | | `http://localhost:8000` | Scraping API endpoint |
| `SCRAPING_SCRAPE_TIMEOUT_S` | | `60` | Scrape tool wait-timeout |
| `SCRAPING_CRAWL_TIMEOUT_S` | | `600` | Crawl tool wait-timeout |
| `SCRAPING_REQUEST_TIMEOUT_S` | | `30` | Individual HTTP request timeout |

## Troubleshooting

**Server not starting:** Check `SCRAPING_API_KEY` is set in MCP client config env. Logs go to stderr — Claude Desktop / Cursor show them in their MCP logs.

**"connection refused":** Verify `SCRAPING_BASE_URL` points to a running Scraping API. For local dev: `docker compose up` in the monorepo.

**Agent receives `{status: "pending", task_id}`:** Timeout exceeded. Agent should call `get_scrape_status(task_id)` (or `get_crawl_status`) to retrieve the result once ready. Scrape typically takes 2-20 seconds; heavy sites (JS-rendered via L3) can take 30-60 seconds.

## License

MIT.

## Links

- [Homepage](https://scraping.io)
- [REST API docs](https://api.scraping.io/docs)
- [Report issues](https://github.com/Nodirbek2001/scraper/issues)
