Metadata-Version: 2.4
Name: agent-web-search
Version: 0.1.0
Summary: Python client for the self-hosted Agent Web Search API — returns cosine-ranked, scrubbed, RAG-ready chunks
Keywords: search,rag,ai-agent,llm,searxng,self-hosted,web-search,tavily-alternative
Author: blueewhitee
Author-email: blueewhitee <[email protected]>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Typing :: Typed
Requires-Dist: httpx>=0.27,<1
Requires-Dist: pydantic>=2.0,<3
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/blueewhitee/agent-web-search
Project-URL: Repository, https://github.com/blueewhitee/agent-web-search
Project-URL: Issues, https://github.com/blueewhitee/agent-web-search/issues
Description-Content-Type: text/markdown

# agent-web-search

[![PyPI](https://img.shields.io/pypi/v/agent-web-search)](https://pypi.org/project/agent-web-search/)
[![Python](https://img.shields.io/pypi/pyversions/agent-web-search)](https://pypi.org/project/agent-web-search/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](#development)

**Python client for the [Agent Web Search](https://github.com/blueewhitee/agent-web-search) API — self-hosted, RAG-ready web search with cosine-ranked chunks.**

Returns scrubbed, chunked, cosine-ranked text from a 6-stage pipeline (SearXNG → fetch → extract → scrub → chunk → embed → rank). Drop-in for Tavily / Exa in self-hosted setups — same shape, no API keys, no usage limits, your data stays on your machine.

## Why this SDK?

| | agent-web-search | Tavily | Exa |
|---|---|---|---|
| **Cost** | Free, self-hosted | $0.05/query | $0.005/query |
| **API key** | None | Required | Required |
| **RAG chunk output** | ✅ 256/512-token with scores | ❌ | ❌ |
| **Prompt scrubbing** | ✅ Built-in | ❌ | ❌ |
| **JS rendering** | ✅ Opt-in | ❌ | ❌ |
| **Type-safe** | ✅ Pydantic v2 | Partial | Partial |
| **Sync + async** | ✅ | ✅ | ✅ |

The `agent-web-search` server is the full pipeline; this client is a thin, type-safe wrapper with sync and async surfaces, Pydantic models for every response, and an exception hierarchy that maps cleanly to OpenAI's SDK pattern.

## Install

```bash
pip install agent-web-search
```

Zero required dependencies beyond `httpx` and `pydantic` (both installed automatically).

## Quick start

### Sync

```python
from agent_web_search import AgentWebSearch

with AgentWebSearch(base_url="http://localhost:8000") as client:
    response = client.search("python asyncio", include_content=True)
    for chunk in response.ranked_chunks:
        print(f"{chunk.score:.3f}  {chunk.source.title}")
        print(f"        {chunk.source.url}")
        print(f"        {chunk.text[:120]}...")
```

### Async

```python
import asyncio
from agent_web_search import AsyncAgentWebSearch

async def main():
    async with AsyncAgentWebSearch(base_url="http://localhost:8000") as client:
        response = await client.search("python asyncio")
        print(response.top_chunk.text if response.top_chunk else "no results")

asyncio.run(main())
```

### Concurrent queries

```python
import asyncio
from agent_web_search import AsyncAgentWebSearch

async def main():
    async with AsyncAgentWebSearch() as client:
        results = await asyncio.gather(
            client.search("python asyncio"),
            client.search("rust borrow checker"),
            client.search("docker healthcheck"),
        )

asyncio.run(main())
```

## API reference

### `AgentWebSearch(base_url=None, *, timeout=30.0, headers=None)` (and async)

Create a client. All arguments are optional — sensible defaults work out of the box if the server is at `http://localhost:8000`.

| Argument | Type | Default | Description |
|---|---|---|---|
| `base_url` | `str` | `AGENT_WEB_SEARCH_BASE_URL` env var, else `http://localhost:8000` | Server root URL |
| `timeout` | `float` | `30.0` seconds | Per-request timeout (covers the full pipeline) |
| `headers` | `dict[str, str]` | `None` | Extra HTTP headers (e.g. future auth tokens) |

Both clients are context managers (sync: `with`, async: `async with`) and can be used without one too (just call `close()` / `await close()` when done).

### `client.search(query, *, render_js=False, include_content=False, categories=None, time_range=None, timeout=None)`

Search the web and return cosine-ranked chunks.

| Argument | Type | Default | Description |
|---|---|---|---|
| `query` | `str` | *required* | 1–1000 characters |
| `render_js` | `bool` | `False` | Use crawl4ai to render JS-heavy pages (slower; opt in only when needed) |
| `include_content` | `bool` | `False` | Fetch, extract, chunk, and rank each page. If `False`, ranks pre-fetched SearXNG snippets |
| `categories` | `list[str]` | `None` | Override intent routing — e.g. `["it"]`, `["news"]`, `["general"]` |
| `time_range` | `str` | `None` | Time filter for news queries: `"day"`, `"week"`, `"month"`, `"year"` |
| `timeout` | `float` | client default | Per-call timeout override |

**Returns:** `SearchResponse` with `query`, `ranked_chunks: list[RankedChunk]`, `unresponsive_engines: list[str]`.

**Raises:** `APITimeoutError`, `APIConnectionError`, `APIError`, `APIResponseError`, or `ValueError` for invalid input.

### `client.health(*, timeout=None)`

Ping the server. Returns `HealthResponse(status="ok")` on success.

### Response models

```python
SearchResponse(
    query="python asyncio",
    ranked_chunks=[
        RankedChunk(
            text="asyncio is a library...",            # ~256-token chunk
            parent_text="asyncio — Asynchronous I/O...", # ~512-token context
            chunk_index=0,
            score=0.87,                                # cosine similarity
            source=ChunkSource(
                url="https://docs.python.org/3/library/asyncio.html",
                title="asyncio — Asynchronous I/O — Python 3.12 documentation",
                searxng_score=1.0,                     # upstream popularity
            ),
        ),
        ...
    ],
    unresponsive_engines=["duckduckgo"],                # engines that timed out
)
```

`SearchResponse` has two convenience properties:
- `response.top_chunk` — the highest-scoring chunk, or `None` if empty
- `response.urls` — the unique source URLs, in ranking order

## Configuration

| Env var | Default | Description |
|---|---|---|
| `AGENT_WEB_SEARCH_BASE_URL` | `http://localhost:8000` | Server URL (overridden by `base_url` kwarg) |
| `AGENT_WEB_SEARCH_TIMEOUT` | `30.0` | Default request timeout in seconds |

## Errors

All exceptions inherit from `AgentWebSearchError`. The hierarchy mirrors the OpenAI SDK so error-handling patterns are portable:

| Exception | When it's raised | Has |
|---|---|---|
| `APITimeoutError` | Request timed out (client-side `timeout` exceeded) | — |
| `APIConnectionError` | Couldn't reach the server (DNS, refused, network) | — |
| `APIError` | Server returned non-2xx (4xx, 5xx) | `status_code`, `body`, `detail` |
| `APIResponseError` | Server returned a body that didn't match the expected schema | — |

```python
from agent_web_search import (
    AgentWebSearch, AgentWebSearchError,
    APIError, APITimeoutError, APIConnectionError,
)

client = AgentWebSearch(base_url="http://localhost:8000")

try:
    response = client.search("python asyncio", timeout=15.0)
except APITimeoutError:
    print("The server is too slow — try increasing timeout")
except APIConnectionError:
    print("Server is not running. Start it with: docker compose up")
except APIError as e:
    print(f"Server returned HTTP {e.status_code}: {e.detail}")
except AgentWebSearchError as e:
    # Catch-all for any other SDK error
    print(f"SDK error: {e}")
```

## Development

```bash
git clone https://github.com/blueewhitee/agent-web-search-sdk
cd agent-web-search-sdk

# Install dev dependencies
uv sync --group dev

# Run tests
uv run pytest -v

# Type check
uv run mypy src  # (if mypy is installed)

# Build sdist + wheel
uv build

# Publish to PyPI (one-time, requires API token)
export UV_PUBLISH_TOKEN="pypi-AgEIcHlwaS5vcmcC..."  # token from pypi.org
uv publish
```

## Publishing (release process)

The SDK uses **PyPI Trusted Publishing via OIDC + GitHub Actions** — no API tokens stored anywhere. Here's the one-time setup:

### One-time setup

1. **Create the GitHub repo** (e.g. `github.com/blueewhitee/agent-web-search`).
2. **Push the SDK code** (this directory) to the repo.
3. **Register a Trusted Publisher on PyPI:**
   - Go to https://pypi.org/manage/project/agent-web-search/settings/publishing/ (or create the project first if it doesn't exist).
   - Add a pending publisher (or normal publisher after the first release):
     - Owner: `blueewhitee`
     - Repository: `agent-web-search`
     - Workflow filename: `publish.yml`
     - Environment name: `pypi`
4. **Optional first release manually** (recommended, to lock in the name immediately):
   ```bash
   # Create a PyPI API token at https://pypi.org/manage/account/token/
   # Scope it to the agent-web-search project only.
   export UV_PUBLISH_TOKEN="pypi-..."
   uv build
   uv publish
   ```
5. **Switch the PyPI publisher from "pending" to "live"** now that the project exists.

### Every release after that

```bash
# Bump version in pyproject.toml and src/agent_web_search/__init__.py
git commit -am "release: v0.2.0"
git tag v0.2.0
git push --tags

# GitHub Actions automatically builds + publishes to PyPI via OIDC.
```

The workflow file at `.github/workflows/publish.yml` handles the rest.

## Architecture

This is intentionally a **thin client** (~250 LOC of production code):

- `src/agent_web_search/client.py` — sync `AgentWebSearch`
- `src/agent_web_search/async_client.py` — async `AsyncAgentWebSearch`
- `src/agent_web_search/models.py` — Pydantic v2 models
- `src/agent_web_search/exceptions.py` — exception hierarchy
- `src/agent_web_search/_base.py` — shared config + httpx→SDK error translation

The models are re-declared in the SDK rather than imported from the server to keep the package self-contained and the dependency surface minimal. The FastAPI server generates an OpenAPI schema at `/openapi.json`; a v0.2 follow-up could add codegen (`datamodel-code-generator`) to keep the two in sync automatically.

## License

[MIT](LICENSE)
