Metadata-Version: 2.5
Name: capturekit-haystack
Version: 0.1.0
Summary: Haystack integration for CaptureKit: fetch web pages as clean Markdown documents for RAG pipelines and agents.
Project-URL: Homepage, https://www.capturekit.dev
Project-URL: Documentation, https://github.com/capturekitdev/capturekit-haystack#readme
Project-URL: Repository, https://github.com/capturekitdev/capturekit-haystack
Project-URL: Issues, https://github.com/capturekitdev/capturekit-haystack/issues
Author-email: CaptureKit <contact@capturekit.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: capturekit,fetcher,haystack,markdown,rag,web-scraping
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: haystack-ai>=2.9
Requires-Dist: httpx<1,>=0.27
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
Requires-Dist: pytest>=8.3; extra == 'test'
Requires-Dist: ruff<1,>=0.8; extra == 'test'
Description-Content-Type: text/markdown

# capturekit-haystack

[Haystack](https://haystack.deepset.ai) integration for [CaptureKit](https://www.capturekit.dev), a web page API that loads pages in a real browser and returns them as clean Markdown.

`CaptureKitFetcher` fetches web pages and returns them as Haystack `Document`s: the content is the page as Markdown, and the metadata holds `url`, `title` and `description`. No HTML converter is needed.

## Installation

```bash
pip install capturekit-haystack
```

Get an API key at [app.capturekit.dev](https://app.capturekit.dev) and set it in the environment:

```bash
export CAPTUREKIT_API_KEY="your-api-key"
```

## Usage

```python
from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

fetcher = CaptureKitFetcher()
docs = fetcher.run(urls=["https://example.com"])["documents"]
print(docs[0].meta["title"], docs[0].content[:200])
```

| Parameter | Default | Description |
| --- | --- | --- |
| `api_key` | `Secret.from_env_var("CAPTUREKIT_API_KEY")` | CaptureKit API key |
| `main_content_only` | `True` | Keep the article or product and drop menus, sidebars and footers |
| `raise_on_failure` | `True` | Raise when a single URL fails; with several URLs, failures are logged and skipped |
| `timeout` | `120` | Seconds to wait for each page |

A failed request raises `CaptureKitError`, a Haystack `ComponentError` with the HTTP status. The API key never appears in the message. `run_async` is available too.

### Index pages for RAG

```python
from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

store = InMemoryDocumentStore()
pipeline = Pipeline()
pipeline.add_component("fetcher", CaptureKitFetcher())
pipeline.add_component("splitter", DocumentSplitter(split_by="word", split_length=200))
pipeline.add_component("writer", DocumentWriter(document_store=store))
pipeline.connect("fetcher.documents", "splitter.documents")
pipeline.connect("splitter.documents", "writer.documents")

pipeline.run({"fetcher": {"urls": ["https://docs.haystack.deepset.ai/docs/intro"]}})
print(store.count_documents())
```

## Development

```bash
pip install -e ".[test]"
pytest
```

## License

MIT
