Metadata-Version: 2.4
Name: webcrawlerapi-langchain
Version: 0.2.0
Summary: LangChain integration for WebCrawlerAPI
Home-page: https://github.com/webcrawlerapi/webcrawlerapi-langchain
Author: WebCrawlerAPI
Author-email: support@webcrawlerapi.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: webcrawlerapi>=2.1.5
Requires-Dist: langchain-core>=0.3.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# WebCrawlerAPI LangChain Integration

[WebcrawlerAPI](https://webcrawlerapi.com/) - is a website to LLM data API. It allows to convert websites and webpages markdown or cleaned content. 

**Pay as you go (No subscription required).**

This package provides LangChain integration for [WebCrawlerAPI](https://webcrawlerapi.com/), allowing you to easily use web crawling capabilities with [LangChain](https://www.langchain.com/) document processing pipeline.

## Installation

Get your [API key](https://webcrawlerapi.com/docs/access-key) first

```bash
pip install webcrawlerapi-langchain
```

## Usage

Requires Python 3.10 or newer. `api_key` and `base_url` fall back to the `WEBCRAWLERAPI_API_KEY` and `WEBCRAWLERAPI_BASE_URL` environment variables when not passed.

### Basic Loading
```python
from webcrawlerapi_langchain import WebCrawlerAPILoader

# Initialize the loader
loader = WebCrawlerAPILoader(
    url="https://example.com",
    api_key="your-api-key",
    output_format="markdown",
    items_limit=10
)

# Load documents
documents = loader.load()

# Use documents in your LangChain pipeline
for doc in documents:
    print(doc.page_content[:100])
    print(doc.metadata)
```

### Async Loading
```python
# Async loading
documents = await loader.aload()
```

### Lazy Loading
Yields each page as soon as it finishes crawling:
```python
# Lazy loading
for doc in loader.lazy_load():
    print(doc.page_content[:100])
```

### Async Lazy Loading
```python
# Async lazy loading
async for doc in loader.alazy_load():
    print(doc.page_content[:100])
```

### Error Handling
Crawling and API failures raise `WebCrawlerAPILoaderError`, which is importable from `webcrawlerapi_langchain`. Invalid parameters raise `ValueError`.
```python
from webcrawlerapi_langchain import WebCrawlerAPILoaderError

try:
    documents = loader.load()
except WebCrawlerAPILoaderError as e:
    print(f"Crawl failed: {e}")
```

## Configuration

The loader accepts the following parameters:

- `url`: The URL to crawl (keyword arguments follow it)
- `api_key`: Your WebCrawlerAPI API key. Defaults to `WEBCRAWLERAPI_API_KEY`
- `base_url`: API base URL. Defaults to `WEBCRAWLERAPI_BASE_URL`
- `output_format`: Content format of each Document: `markdown` (default), `cleaned`, or `html`
- `scrape_type`: Deprecated, use `output_format` instead
- `items_limit`: Maximum number of pages to crawl (default `10`)
- `whitelist_regexp`: Regex pattern for URL whitelist
- `blacklist_regexp`: Regex pattern for URL blacklist
- `main_content_only`: Extract only the main content of each page (default `False`)
- `max_depth`: Maximum crawl depth. `0` is the seed URL only, `1` is the seed URL plus direct links
- `max_age`: Max age in seconds for cached content. `0` always fetches fresh content
- `respect_robots_txt`: Whether to respect robots.txt (default `False`)
- `keep_query_params`: Keep URL query params when deduplicating links
- `max_polls`: Maximum number of job status checks before giving up (default `100`)

Document metadata keys: `url`, `title`, `status_code`, `created_at`, `referred_url`, `depth`, `cost`, `job_id`, `item_id`.

### Links
- [WebCrawlerAPI Python SDK](https://github.com/WebCrawlerAPI/webcrawlerapi-python-sdk)
- [WebCrawlerAPI Documentation](https://webcrawlerapi.com/docs/getting-started)
- [WebCrawlerAPI API Reference](https://webcrawlerapi.com/docs/api-reference)

If you need help with integration feel free to [contact us](support@webcrawlerapi.com).

## License

MIT License 
