Metadata-Version: 2.5
Name: langchain-scrapely
Version: 0.1.0
Summary: LangChain integration for Scrape.ly: run Actors, load their results as documents, and give agents Scrape.ly tools.
Project-URL: Homepage, https://scrape.ly
Project-URL: Documentation, https://scrape.ly/docs/langchain
Project-URL: Repository, https://github.com/scrapely-app/langchain-scrapely
Author: TeraShift GmbH
License-Expression: MIT
License-File: LICENSE
Keywords: agents,langchain,langgraph,rag,scrapely,scraping,tools
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Requires-Dist: langchain-core<2,>=0.3.15
Requires-Dist: scrapely-client-python<2,>=1.0.8
Description-Content-Type: text/markdown

# langchain-scrapely

[LangChain](https://www.langchain.com) and [LangGraph](https://www.langchain.com/langgraph) integration for [Scrape.ly](https://scrape.ly): run Actors, load their results as documents for RAG, and give agents Scrape.ly tools.

```bash
pip install langchain-scrapely
export SCRAPELY_TOKEN=your-api-key
```

Create an API key in the Scrape.ly dashboard under **Settings → API keys**. Everything below reads it from `SCRAPELY_TOKEN`, or takes it as `scrapely_token=...`.

## Load results as documents

`ScrapelyWrapper` runs an Actor, waits for it to finish, and returns a loader for its results. You decide how each result becomes a `Document`.

```python
from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper

loader = ScrapelyWrapper().call_actor(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get('text') or '', metadata={'source': item.get('url')}
    ),
)
docs = loader.load()
```

- `call_actor_task(task_id, task_input, ...)` runs a saved task instead. Fields in `task_input` override its saved input.
- `acall_actor` and `acall_actor_task` are the async versions.
- `build`, `memory_mbytes` and `timeout_secs` set the run options.
- A run that does not succeed raises `RuntimeError`.

To load results that already exist, use `ScrapelyDatasetLoader`:

```python
from langchain_scrapely import ScrapelyDatasetLoader

loader = ScrapelyDatasetLoader(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(page_content=item.get('text') or ''),
)
docs = loader.load()  # or lazy_load(), aload(), alazy_load()
```

## Tools for agents

### Any Actor as a tool

`ScrapelyActorsTool` turns one Actor into a tool. Its name, description and input come from the Actor, so the agent knows what it does and how to fill in its input.

```python
from langchain_scrapely import ScrapelyActorsTool

tool = ScrapelyActorsTool('username/actor-name')
result = tool.invoke({'run_input': {'query': 'coffee shops in Berlin'}})
```

### General tools

| Tool | Name | What it does |
|---|---|---|
| `ScrapelyRunActorTool` | `scrapely_run_actor` | Runs an Actor and returns the run, including its `dataset_id`. |
| `ScrapelyRunActorAndGetDatasetTool` | `scrapely_run_actor_and_get_dataset` | Runs an Actor and returns its results. |
| `ScrapelyGetDatasetItemsTool` | `scrapely_get_dataset_items` | Reads items from a dataset. |
| `ScrapelyRunTaskTool` | `scrapely_run_task` | Runs a saved task and returns the run. |
| `ScrapelyRunTaskAndGetDatasetTool` | `scrapely_run_task_and_get_dataset` | Runs a saved task and returns its results. |

`SCRAPELY_CORE_TOOLS` holds all five as classes:

```python
from langchain_scrapely import SCRAPELY_CORE_TOOLS

tools = [cls() for cls in SCRAPELY_CORE_TOOLS]
```

The tools that return results answer with JSON: `{"run": {id, status, status_message, dataset_id, started_at, finished_at}, "items": [...]}`. `scrapely_run_actor` and `scrapely_run_task` return only `run`, and `scrapely_get_dataset_items` returns `"run": null`. Errors, including invalid arguments, go back to the agent as text instead of stopping it.

Limits on what the agent may ask for, set when you create a tool:

| Option | Default | What it limits |
|---|---|---|
| `max_timeout_secs` | `600` | Longest run the agent may start. |
| `max_memory_mbytes` | `32768` | Most memory per run. Requests are rounded down to a power of two; anything under 128 MB gets 128 MB. |
| `max_items` | `1000` | Most results returned at once. |

### With LangGraph

`create_agent` builds a LangGraph agent. It is in the `langchain` package, and the model below needs `langchain-openai` (`pip install langchain langchain-openai`):

```python
from langchain.agents import create_agent
from langchain_scrapely import ScrapelyActorsTool, ScrapelyGetDatasetItemsTool

agent = create_agent(
    'openai:gpt-4o',
    tools=[ScrapelyActorsTool('username/actor-name'), ScrapelyGetDatasetItemsTool()],
)
agent.invoke({'messages': [{'role': 'user', 'content': 'Find the top coffee shops in Berlin'}]})
```

Give an agent only the tools it needs: fewer tools means better choices and fewer tokens.

## Docs

[scrape.ly/docs/langchain](https://scrape.ly/docs/langchain) and [scrape.ly/docs/langgraph](https://scrape.ly/docs/langgraph).

## License

[MIT](LICENSE)
