Metadata-Version: 2.5
Name: skim-haystack
Version: 0.1.4
Summary: Haystack integration for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).
Project-URL: Homepage, https://skim402.com
Project-URL: Documentation, https://skim402.com/docs
Project-URL: Repository, https://github.com/JessieJanie/skim402
Project-URL: x402 protocol, https://x402.org
Author-email: Skim <hello@skim402.com>
License: MIT
License-File: LICENSE
Keywords: agent,ai,deepset,haystack,haystack-ai,llm,markdown,rag,reader,skim,web-scraping,x402
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: eth-account>=0.13.0
Requires-Dist: haystack-ai>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: x402[evm]>=2.0.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

# skim-haystack

**Give your Haystack pipelines the ability to read any URL — clean Markdown, ~4x smaller than raw HTML. No ads, no nav, no boilerplate.**

[![PyPI version](https://img.shields.io/pypi/v/skim-haystack.svg)](https://pypi.org/project/skim-haystack/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

`skim-haystack` is the official [Haystack](https://haystack.deepset.ai) integration for [Skim](https://skim402.com) — the clean reader API for AI agents. It provides one component, `SkimReader`, that fetches any web page and returns it as a Haystack `Document` (clean Markdown in `content`, structured metadata in `meta`). Output is ~4x smaller than raw HTML — your pipeline processes more pages with fewer tokens.

**Two ways to pay:** card plan with API key (free tier: 1,000 reads/month — [skim402.com/pricing](https://skim402.com/pricing)) or x402 wallet pay-per-call ($0.002 USDC on Base, no account needed).

> **See it before you wire it:** [try Skim free in your browser](https://freeskims.skim402.com) — 10 free skims a day, no wallet, no signup. Paste a URL, see exactly what your agent gets back.

---

## Install

```bash
pip install skim-haystack
```

---

## Quickstart (60 seconds)

### 1. Get a free API key

Go to **[skim402.com/pricing](https://skim402.com/pricing)**, sign up for the free plan (1,000 reads/month), and copy your `sk402_...` key.

### 2. Set the env var

```bash
export SKIM_API_KEY=sk402_your_key_here
```

### 3. Use it

```python
from skim_haystack import SkimReader

reader = SkimReader()  # reads SKIM_API_KEY from the environment

result = reader.run(urls="https://en.wikipedia.org/wiki/HTTP_402")
print(result["documents"][0].content)
```

---

## Alternative: pay per call with a crypto wallet

If you prefer x402 wallet pay-per-call instead of a card plan:

```bash
export SKIM_WALLET_PRIVATE_KEY=0xYOUR_BASE_WALLET_PRIVATE_KEY
```

Fund a dedicated Base wallet with a small USDC balance. Each read costs $0.002 on Base. Full setup guide: **<https://skim402.com/wallet>**.

> **Use a fresh wallet, not your personal one.** This wallet's private key signs payment authorizations on your machine — treat it like a hot wallet for paying $0.002 tolls, not a savings account.

Or configure explicitly with Haystack Secrets:

```python
from haystack.utils import Secret
from skim_haystack import SkimReader

reader = SkimReader(
    private_key=Secret.from_token("0x..."),
    max_price_usd=0.005,
    include_metadata=True,
)
```

---

## Use it in a pipeline

`SkimReader` is a standard Haystack component, so it drops straight into a `Pipeline`. Here it fetches a page and feeds the cleaned Markdown into a prompt:

```python
from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from skim_haystack import SkimReader

pipe = Pipeline()
pipe.add_component("reader", SkimReader())
pipe.add_component("prompt", PromptBuilder(
    template="Summarize this article in 5 bullets:\n\n{{ documents[0].content }}"
))
pipe.add_component("llm", OpenAIGenerator(model="gpt-4o-mini"))

pipe.connect("reader.documents", "prompt.documents")
pipe.connect("prompt.prompt", "llm.prompt")

result = pipe.run({"reader": {"urls": "https://en.wikipedia.org/wiki/HTTP_402"}})
print(result["llm"]["replies"][0])
```

---

## Output shape

`SkimReader.run(...)` returns `{"documents": [Document, ...]}` — one `Document` per URL:

- `Document.content` — the cleaned article body in Markdown.
- `Document.meta` — always includes `source` (the URL), plus page metadata (`title`, `byline`, `publishedAt`, `lang`, `excerpt`, ...) unless `include_metadata=False`.

---

## Configuration

`SkimReader` takes the following parameters:

| Parameter          | Default                                              | Notes                                                                                                                          |
| ------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `api_key`          | `Secret.from_env_var("SKIM_API_KEY", strict=False)` | Card-plan API key. Get one free at skim402.com/pricing. Takes priority over `private_key`.                                    |
| `private_key`      | `Secret.from_env_var("SKIM_WALLET_PRIVATE_KEY", strict=False)` | Wallet lane only. Hex Base wallet private key. Ignored when `api_key` resolves.                                  |
| `base_url`         | `https://skim402.com`                               | Override the API base URL. For self-hosting or local development.                                                             |
| `max_price_usd`    | `0.01`                                              | Wallet lane only. Hard cap on per-call price in USD. Skim is `$0.002`/call.                                                   |
| `include_metadata` | `True`                                              | Populate each `Document`'s `meta` with page metadata.                                                                         |
| `timeout`          | `60`                                                | Per-request timeout in seconds.                                                                                               |

The component supports pipeline serialization (`to_dict`/`from_dict`). When the key comes from an environment variable (the default), it is stored as a *reference* to that variable name — never the raw value. An inline `Secret.from_token("...")` is runtime-only and will not serialize.

---

## Security

- **No outbound telemetry from this package.** `skim-haystack` only talks to `skim402.com` (or whatever you set as `base_url`). No analytics, no error reporting, no phone-home.
- **Wallet lane:** the private key only signs payment authorizations locally — it never leaves your machine.

---

## Links

- **Skim website** — <https://skim402.com>
- **Pricing & free key** — <https://skim402.com/pricing>
- **Wallet setup guide** — <https://skim402.com/wallet>
- **API docs** — <https://skim402.com/docs>
- **GitHub** — <https://github.com/JessieJanie/skim402>

---

## License

MIT
