Metadata-Version: 2.5
Name: llama-index-readers-scrapely
Version: 0.1.0
Summary: LlamaIndex readers for Scrape.ly: run Actors or read datasets and get Documents back.
Project-URL: Homepage, https://scrape.ly
Project-URL: Documentation, https://scrape.ly/docs/llama-index
Project-URL: Repository, https://github.com/scrapely-app/llama-index-readers-scrapely
Author: TeraShift GmbH
License-Expression: MIT
License-File: LICENSE
Keywords: llama-index,llamaindex,rag,reader,scrapely,scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Requires-Dist: llama-index-core<0.15,>=0.13.0
Requires-Dist: scrapely-client-python<2,>=1.0.8
Description-Content-Type: text/markdown

# llama-index-readers-scrapely

[LlamaIndex](https://www.llamaindex.ai) readers for [Scrape.ly](https://scrape.ly): run an Actor, or read an existing dataset, and get `Document`s back for your index.

```bash
pip install llama-index-readers-scrapely
export SCRAPELY_TOKEN=your-api-key
```

Create an API key in the Scrape.ly dashboard under **Settings → API keys**. The readers read it from `SCRAPELY_TOKEN`, or take it as `scrapely_token=...`.

## Run an Actor

```python
from llama_index.core import Document, VectorStoreIndex
from llama_index.readers.scrapely import ScrapelyActor

docs = ScrapelyActor().load_data(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        text=item.get('text') or '', metadata={'url': item.get('url')}
    ),
)
index = VectorStoreIndex.from_documents(docs)
```

The index and query engine use OpenAI by default: `pip install llama-index` (or `llama-index-embeddings-openai llama-index-llms-openai`) and set `OPENAI_API_KEY`.

`build`, `memory_mbytes` and `timeout_secs` set the run options. A run that does not succeed raises `RuntimeError`.

## Read an existing dataset

```python
from llama_index.readers.scrapely import ScrapelyDataset

docs = ScrapelyDataset().load_data(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(text=item.get('text') or ''),
)
```

`dataset_mapping_function` decides how each result becomes a `Document`, and must return one.

## Docs

[scrape.ly/docs/llama-index](https://scrape.ly/docs/llama-index)

## License

[MIT](LICENSE)
