Metadata-Version: 2.5
Name: xdive
Version: 1.5.0
Summary: Python SDK for XDive: scrape, crawl, map, search and research the web with citations, for AI agents.
Project-URL: Homepage, https://crawler.mehdisadeghian.com
Project-URL: Documentation, https://crawler.mehdisadeghian.com/docs/sdk-python
Author: XDive
License: MIT
License-File: LICENSE
Keywords: agents,crawler,langchain,llm,markdown,mcp,rag,research,search,web scraping,xdive
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx<1,>=0.25
Provides-Extra: dev
Requires-Dist: langchain-core>=0.2; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == 'langchain'
Provides-Extra: llama-index
Requires-Dist: llama-index-core>=0.10; extra == 'llama-index'
Description-Content-Type: text/markdown

# xdive

Python client for [XDive](https://crawler.mehdisadeghian.com): scrape pages
to Markdown, crawl and map sites, search the web, and answer questions with
verified citations. Synchronous and asynchronous clients, typed errors,
automatic retries, and LangChain and LlamaIndex integrations.

Full documentation: https://crawler.mehdisadeghian.com/docs/sdk-python

## Install

```bash
pip install xdive
pip install "xdive[langchain]"     # LangChain tools and document loader
pip install "xdive[llama-index]"   # LlamaIndex reader
```

Requires Python 3.9 or later. The only runtime dependency is `httpx`.

## Quickstart

```python
from xdive import XDive, page_markdown

with XDive() as xdive:  # reads XDIVE_API_KEY
    page = xdive.scrape("https://example.com", formats=["markdown", "links"])
    print(page_markdown(page))
    site = xdive.crawl("https://docs.example.com", max_pages=50)  # waits for the crawl
    answer = xdive.research("What does PEP 639 change about license metadata?")
    print(answer["answer"])
    for citation in answer["citations"]:
        print(citation["number"], citation["url"])
```

Create an API key under **API Keys** in the dashboard. A key belongs to one
organization; the client looks the organization up once through
`GET /whoami`, so the key is all you configure.

## Configuration

| Argument | Environment variable | Default | Description |
|---|---|---|---|
| `api_key` | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`) | (required) | Organization API key (`ck_live_…`). |
| `base_url` | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`) | `https://crawler.mehdisadeghian.com/api` | API base URL, including `/api`. |
| `org_id` | | discovered | Skip the `/whoami` lookup when you know the organization id. |
| `timeout` | | `120.0` | Per-request timeout in seconds. |
| `max_retries` | | `3` | Retries of overload responses: `429`, and `503` with code `SERVER_BUSY`. |

A missing key raises `ValueError` when the client is created.

## Methods

`XDive` and `AsyncXDive` have the same methods; on `AsyncXDive` each is
a coroutine. Keyword options are passed through as the API's JSON fields.

| Method | API call | Returns |
|---|---|---|
| `whoami()` | `GET /whoami` | The key's organization, scopes and plan. |
| `scrape(url, formats=None, **options)` | `POST /orgs/{id}/scrape` | The page object (`data`). |
| `crawl(url, *, timeout=600, poll_interval=2, **options)` | `POST /orgs/{id}/crawl`, then polls | The finished crawl record with `results`. |
| `start_crawl(url, **options)` | `POST /orgs/{id}/crawl` | The crawl id. |
| `get_crawl(crawl_id)` | `GET /orgs/{id}/crawls/{crawlId}` | The crawl record. |
| `wait_for_crawl(crawl_id, *, timeout=600, poll_interval=2)` | polls `GET /orgs/{id}/crawls/{crawlId}` | The finished crawl record. |
| `cancel_crawl(crawl_id)` | `DELETE /orgs/{id}/crawls/{crawlId}` | `None`. |
| `crawl_errors(crawl_id)` | `GET /orgs/{id}/crawls/{crawlId}/errors` | Failed pages (`errors`) and `robots_blocked` URLs. |
| `batch_scrape(urls, *, wait=True, timeout=600, **options)` | `POST /orgs/{id}/batch/scrape` | The finished record, or the job id with `wait=False`. |
| `map(url, *, search=None, limit=None, max_depth=None)` | `POST /orgs/{id}/map` | A list of URLs. |
| `search(query, **options)` | `POST /orgs/{id}/search` | The search response with `results`. |
| `research(query, *, depth="quick", output=None, schema=None, **options)` | `POST /orgs/{id}/research` | `answer` or `data`, `citations`, `sources`, `confidence`, `usage`. |
| `extract(urls, *, prompt=None, schema=None, **options)` | `POST /orgs/{id}/extract` | The extracted data. |
| `llms_txt(url, *, max_urls=None, full_text=False)` | `POST /orgs/{id}/llmstxt` | `llms_txt` (and `llms_full_txt`). |

`XDive.org_id` is a property; on `AsyncXDive` it is a coroutine,
`await client.org_id()`.

`formats` takes the API's format names (`markdown`, `text`, `html`,
`rawHtml`, `links`, `images`, `json`, `summary`, `question`, `highlights`,
`branding`, `audio`, `screenshot`, `product`, `menu`) or dicts such as
`{"type": "json", "schema": {...}}`. `page_markdown(page)` returns a page's
Markdown whichever field carries it (`artifacts.markdown.content` or
`markdown`).

## Async

```python
import asyncio
from xdive import AsyncXDive

async def main():
    async with AsyncXDive() as client:
        pages = await asyncio.gather(*(client.scrape(u) for u in ["https://a.example", "https://b.example"]))

asyncio.run(main())
```

Close clients you do not use as context managers: `xdive.close()` or
`await client.aclose()`.

## Errors and retries

| Exception | When | Attributes |
|---|---|---|
| `XDiveError` | Any API error (status 400 and above). Base class of the others. | `status`, `message`, `code` |
| `RateLimited` | Still overloaded (`429`, or `503` with code `SERVER_BUSY`) after `max_retries` retries. | `retry_after` (seconds) |
| `QuotaExceeded` | `402`: the organization's monthly quota is used up. | |
| `CrawlTimeout` | `crawl`, `wait_for_crawl` or `batch_scrape` waited longer than `timeout`. The job keeps running on the server. | `crawl_id` |

Only overload is retried: a `429` (organization rate or concurrency limit)
and a `503` whose JSON body has the code `SERVER_BUSY` (server at capacity).
The client waits the number of seconds in the response's `Retry-After`
header (1 second when absent) before each retry. Every other error, including
any other `503` (for example a feature the server has not configured), raises
`XDiveError` at once with the API's `status`, `message` and `code`. Network
errors and timeouts surface as `httpx` exceptions.

Crawl polling sends `If-None-Match`, so an unchanged status comes back as an
empty `304`, and waits the interval the server sets in `Retry-After`,
falling back to `poll_interval`.

## Integrations

```python
from xdive.langchain import XDiveLoader, XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool

tools = [XDiveSearchTool(), XDiveScrapeTool(), XDiveResearchTool()]
docs = XDiveLoader("https://docs.example.com", mode="crawl", max_pages=100).load()
```

```python
from xdive.llama_index import XDiveReader

documents = XDiveReader().load_data("https://docs.example.com", mode="crawl", max_pages=100)
```

`XDiveSearchTool` returns each result's query-relevant passages rather than
whole pages, which keeps an agent's context small. See the
[LangChain](https://crawler.mehdisadeghian.com/docs/langchain) and
[LlamaIndex](https://crawler.mehdisadeghian.com/docs/llamaindex) guides.

## Self-hosted

```python
xdive = XDive(api_key="ck_live_…", base_url="https://your-host/api")
```

or set `XDIVE_BASE_URL=https://your-host/api`.

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT. The license text is in the `LICENSE` file included in the package.
