Metadata-Version: 2.4
Name: bytecrawl
Version: 1.0.0
Summary: Focused crawling for LLM data collection: Shark-Search and OPIC crawlers plus a single scraping API (static HTML, dynamic JS, APIs, session login, Markdown for LLMs) on a 3-dependency core.
Author-email: abrahamperz <aperez@adokmx.com>
License: MIT
Project-URL: Homepage, https://github.com/abrahamperz/ByteCrawl
Project-URL: Repository, https://github.com/abrahamperz/ByteCrawl
Keywords: scraping,crawler,focused-crawling,shark-search,opic,pagerank,beautifulsoup,playwright,markdown,llm
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Intended Audience :: Developers
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.11
Requires-Dist: lxml>=4.9
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == "browser"
Provides-Extra: llm
Requires-Dist: markdownify>=0.11; extra == "llm"
Requires-Dist: trafilatura>=1.6; extra == "llm"
Provides-Extra: mcp
Requires-Dist: mcp>=2; python_version >= "3.10" and extra == "mcp"
Requires-Dist: markdownify>=0.11; extra == "mcp"
Requires-Dist: trafilatura>=1.6; extra == "mcp"
Provides-Extra: all
Requires-Dist: playwright>=1.40; extra == "all"
Requires-Dist: markdownify>=0.11; extra == "all"
Requires-Dist: trafilatura>=1.6; extra == "all"
Requires-Dist: mcp>=2; python_version >= "3.10" and extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# ByteCrawl

[![CI](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml/badge.svg)](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/bytecrawl)](https://pypi.org/project/bytecrawl/)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/bytecrawl/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Give your AI agent focused web crawling.** ByteCrawl is an MCP server (and a
small Python library) that doesn't just scrape a page — it crawls a whole site
and returns the pages *most relevant* to your topic first, using Shark-Search
and OPIC in pure Python.

- **Webpage**: https://bytecrawl.vercel.app/
- **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp

## Quick start (MCP — nothing to install)

Point any MCP-capable agent (Claude Code, Claude Desktop, Cursor...) at the
hosted endpoint:

```bash
claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp
```

Now the agent has four tools:

| Tool | What it does |
|---|---|
| `focused_crawl` | Crawl a site, rank pages by relevance to a query (Shark-Search / OPIC / BFS) |
| `fetch_markdown` | One page → clean Markdown (5–10× fewer tokens than raw HTML) |
| `extract` | Structured records via CSS selectors |
| `fetch_json_api` | Hit a hidden JSON API |

The hosted server is static-only, rate-limited per IP, caps crawls at 10 pages,
and refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites,
run it locally:

```bash
pip install bytecrawl[mcp]
claude mcp add bytecrawl -- bytecrawl-mcp                 # full power, on your machine
pip install bytecrawl[browser] && playwright install chromium   # + JS rendering
```

## Why focused crawling?

Most crawlers visit pages in whatever order they find them. With a limited
request budget, order is everything — Shark-Search chases the branches that
smell like your query and lets the rest decay, so 100 requests get you the 100
*most useful* pages, not the 100 closest to the seed.

```python
from bytecrawl import SharkSearch

result = SharkSearch(query="vector databases").crawl(
    "https://example.com", max_pages=100)

for page in result.top(10):
    print(f'{page["relevance"]:.3f}  {page["url"]}')
```

- **BFS** — level by level, closest to the seed first.
- **Shark-Search** (Hersovici et al., 1998) — topical best-first; links inherit
  their parent's relevance with decay.
- **OPIC** (Abiteboul et al., 2003) — live PageRank via "cash" flow, no full
  graph needed (a `pagerank()` implementation is included to compare against).

Versus the alternatives: Scrapy is a framework you wire up yourself, Firecrawl
is a paid SaaS — ByteCrawl is a plain library with a 3-package core and these
frontier strategies built in.

## Library API

```python
from bytecrawl import Scraper

bot = Scraper()
page = bot.fetch("https://books.toscrape.com")   # auto: static, browser fallback
books = page.extract("article.product_pod",
                     {"title": "h3 a::attr(title)", "price": "p.price_color::text"})
page.markdown()   # clean Markdown for LLMs   ·   page.tokens()   # token estimate
```

```python
bot.static(url)                                   # plain HTML
bot.api(url, params={...})                        # hidden JSON API
bot.browser(url, wait="div.results")              # JS via Playwright
bot.crawl(url, item="article", fields={...},
          next_page="li.next a::attr(href)")      # pagination
bot.session().login(url, data, csrf_field="csrf_token")   # authenticated
```

## Install

```bash
pip install bytecrawl            # slim core (requests + beautifulsoup4 + lxml)
pip install bytecrawl[llm]       # + Markdown for LLMs
pip install bytecrawl[browser]   # + Playwright
pip install bytecrawl[mcp]       # + local MCP server
pip install bytecrawl[all]
```

## Learn each scraping technique

A guided walkthrough with a runnable example against a practice site:
[static HTML](docs/01-html-estatico.md) ·
[dynamic JS](docs/02-js-dinamico.md) ·
[hidden APIs](docs/03-api-oculta.md) ·
[pagination](docs/04-crawling-paginacion.md) ·
[login](docs/05-login-sesion.md) ·
[graph crawling](docs/06-crawling-grafos.md) ·
[Markdown for LLMs](docs/markdown-llms.md) ·
[ethics](docs/nota-etica.md)

## Contributing

```bash
pip install -e ".[llm,dev,mcp]"
pytest              # 109 tests, no network required
pytest -m live      # + live browser tests (needs the browser extra)
ruff check bytecrawl tests
```

Scrape responsibly: respect `robots.txt`, terms of service and rate limits.
ByteCrawl ships with a configurable delay between requests.

## License

MIT — see [LICENSE](LICENSE).
