Metadata-Version: 2.4
Name: trawl-search
Version: 0.1.0
Summary: Self-hosted, Tavily-compatible search API for local development. Point your Tavily SDK at localhost and stop burning credits.
Project-URL: Homepage, https://github.com/adityaparab/trawl
Project-URL: Issues, https://github.com/adityaparab/trawl/issues
Author-email: Aditya Parab <mradityaparab@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,llm,rag,search,search-api,searxng,self-hosted,tavily
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.10
Requires-Dist: ddgs>=9.0
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.27
Requires-Dist: lxml>=5.2
Requires-Dist: pydantic-settings>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: trafilatura>=2.0
Requires-Dist: uvicorn[standard]>=0.30
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# Trawl 🎣

[![CI](https://github.com/adityaparab/trawl/actions/workflows/ci.yml/badge.svg)](https://github.com/adityaparab/trawl/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/trawl-search)](https://pypi.org/project/trawl-search/)
[![Python](https://img.shields.io/pypi/pyversions/trawl-search)](https://pypi.org/project/trawl-search/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/adityaparab/trawl/blob/main/LICENSE)

**A self-hosted, Tavily-compatible search API for local development.**

Point your existing Tavily SDK at `localhost` and keep building — no credits burned, no monthly limits, no code changes beyond one base-URL argument. When you ship, flip the base URL back and everything behaves the same.

```python
from tavily import TavilyClient

client = TavilyClient(api_key="tvly-anything", api_base_url="http://localhost:7300")
client.search("what is the latest python release?", include_answer=True)
# same request/response shapes, same field names, same error envelope
```

## Why

Agent projects hammer their search tool during development — every test run, every eval loop, every "let me just try this prompt again". Paying per-request (or exhausting a free tier) to debug your own code is silly. Trawl reimplements Tavily's full HTTP API on top of infrastructure you run yourself, so development traffic is free and unlimited, and switching between Trawl and the real Tavily is a one-line change.

## What you get

| Endpoint | Status | Notes |
|---|---|---|
| `POST /search` | ✅ | topics, depths, domain filters, time ranges, images, answers, raw content, favicons, usage, auto_parameters, exact_match |
| `POST /extract` | ✅ | markdown/text, query-focused chunking, images, favicons, failed_results |
| `POST /crawl` | ✅ | BFS with regex path/domain selectors, robots.txt, LLM-driven `instructions` |
| `POST /map` | ✅ | URL discovery |

- **Wire-compatible**: same request fields, same response shapes (`follow_up_questions: null` and all), same `{"detail": {"error": ...}}` error envelope both official SDKs parse, HTTP 400/401/403 semantics, Bearer *and* legacy body `api_key` auth, keyless mode header.
- **No mandatory dependencies on anything paid**: search comes from [SearXNG](https://github.com/searxng/searxng) (bundled, self-hosted, aggregates Google/Bing/DDG) or DuckDuckGo scraping as a zero-setup fallback.
- **Optional local LLM**: point `TRAWL_LLM_BASE_URL` at Ollama / LM Studio / any OpenAI-compatible endpoint to get real generated answers, semantic relevance scores, and instruction-guided crawls. Without one, Trawl degrades gracefully (extractive answers, BM25 scores).
- **Optional response cache**: `TRAWL_CACHE_ENABLED=true` makes repeated dev-loop queries instant.

## Quick start

### Docker (recommended — everything included)

```bash
git clone https://github.com/adityaparab/trawl && cd trawl
docker compose up -d
curl -s -X POST http://localhost:7300/search \
  -H "Authorization: Bearer tvly-dev-anything" \
  -H "Content-Type: application/json" \
  -d '{"query": "self-hosted search engines"}' | python -m json.tool
```

That starts Trawl on **:7300** and a pre-configured SearXNG on **:8080**.

### pip / uv (Trawl only)

```bash
pip install trawl-search        # or: uv tool install trawl-search
trawl serve
```

Without a SearXNG instance, set `TRAWL_SEARCH_PROVIDER=ddgs` for zero-infrastructure DuckDuckGo scraping (fine for light use), or start just SearXNG from this repo's compose file: `docker compose up -d searxng`.

Check your setup at any time:

```bash
trawl check
```

## Pointing your app at Trawl

The official SDKs accept a base-URL override, so **no code changes** are needed beyond configuration:

**Python** ([tavily-python](https://github.com/tavily-ai/tavily-python))

```python
TavilyClient(api_key=os.environ["TAVILY_API_KEY"], api_base_url=os.getenv("TAVILY_BASE_URL", "https://api.tavily.com"))
```

**JavaScript** ([@tavily/core](https://github.com/tavily-ai/tavily-js))

```js
tavily({ apiKey: process.env.TAVILY_API_KEY, apiBaseURL: process.env.TAVILY_BASE_URL })
```

**LangChain** ([langchain-tavily](https://github.com/tavily-ai/langchain-tavily)) — all four tool wrappers accept the override too:

```python
TavilySearch(max_results=5, api_base_url=os.getenv("TAVILY_BASE_URL"))
```

Set `TAVILY_BASE_URL=http://localhost:7300` in development, leave it unset in production. Any non-empty API key works against Trawl by default, so your real `TAVILY_API_KEY` — or a fake `tvly-dev-local` — both pass. Runnable versions of all of these are in [examples/](https://github.com/adityaparab/trawl/tree/main/examples), and [docs/integrations.md](https://github.com/adityaparab/trawl/blob/main/docs/integrations.md) covers frameworks without a base-URL option.

## Configuration

Copy [.env.example](https://github.com/adityaparab/trawl/blob/main/.env.example) to `.env` next to where you run `trawl serve` (or the compose file). Everything is optional; defaults work out of the box. Highlights:

| Variable | Default | What it does |
|---|---|---|
| `TRAWL_PORT` | `7300` | API port |
| `TRAWL_SEARCH_PROVIDER` | `searxng` | `searxng` or `ddgs` |
| `TRAWL_SEARXNG_URL` | `http://localhost:8080` | your SearXNG instance |
| `TRAWL_PROVIDER_FALLBACK` | `true` | retry via DDGS if SearXNG is down |
| `TRAWL_API_KEYS` | *(empty)* | comma-separated allowlist; empty accepts any non-empty key |
| `TRAWL_LLM_BASE_URL` | *(empty)* | OpenAI-compatible endpoint for answers/instructions |
| `TRAWL_LLM_MODEL` | *(empty)* | chat model name, e.g. `llama3.2` |
| `TRAWL_EMBEDDING_MODEL` | *(empty)* | if set, semantic scoring via `/embeddings` |
| `TRAWL_CACHE_ENABLED` | `false` | TTL response cache for dev loops |

With Ollama, a full local setup is:

```bash
ollama pull llama3.2 && ollama pull nomic-embed-text
cat >> .env <<'EOF'
TRAWL_LLM_BASE_URL=http://localhost:11434/v1
TRAWL_LLM_MODEL=llama3.2
TRAWL_EMBEDDING_MODEL=nomic-embed-text
EOF
```

## Honest differences from Tavily

Trawl is a faithful *API* clone, not a search-quality clone. Know what you're trading:

- **Result quality/order differs.** Tavily runs a purpose-built index; Trawl aggregates general-purpose engines and re-ranks locally (BM25, or embeddings when configured). Fine for development; validate ranking-sensitive logic against real Tavily before shipping.
- **`topic=finance` behaves like `general`.** There's no specialized finance index to draw from.
- **`search_depth: fast/ultra-fast` behave like `basic`.** They exist so requests don't fail.
- **`auto_parameters` is a simple heuristic**, not Tavily's learned tuning. It's echoed back in the response the same way.
- **Scores are comparable within a response, not across services.** Don't hard-code thresholds tuned on Tavily scores.
- **`usage.credits` reports what Tavily *would* charge** (approximately, for crawl), so client-side accounting keeps working — no actual metering happens.
- **News dates** depend on what the upstream engine exposes; undated results are not filtered out by `start_date`/`end_date`.

## Documentation

| | |
|---|---|
| [docs/api-reference.md](https://github.com/adityaparab/trawl/blob/main/docs/api-reference.md) | every endpoint, parameter, response shape, and error code |
| [docs/configuration.md](https://github.com/adityaparab/trawl/blob/main/docs/configuration.md) | all `TRAWL_*` variables + Ollama/LM Studio/caching recipes |
| [docs/integrations.md](https://github.com/adityaparab/trawl/blob/main/docs/integrations.md) | wiring tavily-python, @tavily/core, LangChain, raw HTTP |
| [docs/compatibility.md](https://github.com/adityaparab/trawl/blob/main/docs/compatibility.md) | exact parity vs. approximations vs. intentional differences |
| [docs/providers.md](https://github.com/adityaparab/trawl/blob/main/docs/providers.md) | how search backends work / how to add one |
| [docs/deployment.md](https://github.com/adityaparab/trawl/blob/main/docs/deployment.md) | compose, systemd, hardening beyond localhost |
| [examples/](https://github.com/adityaparab/trawl/tree/main/examples) | runnable snippets for every SDK |

Interactive OpenAPI docs are at `http://localhost:7300/docs` while serving.

## Development & contributing

```bash
uv venv && uv pip install -e ".[dev]"
.venv/bin/pytest            # offline test suite (mocked providers & pages)
.venv/bin/ruff check src tests
trawl serve --reload
```

See [CONTRIBUTING.md](https://github.com/adityaparab/trawl/blob/main/CONTRIBUTING.md) for the parity-first ground rules and project layout, and [docs/providers.md](https://github.com/adityaparab/trawl/blob/main/docs/providers.md) for the most-wanted contribution: new search providers (Brave, Kagi, Google CSE, ...). Security reports: [SECURITY.md](https://github.com/adityaparab/trawl/blob/main/SECURITY.md) — please read its threat-model section before exposing Trawl beyond localhost. Release history: [CHANGELOG.md](https://github.com/adityaparab/trawl/blob/main/CHANGELOG.md).

## License

[MIT](https://github.com/adityaparab/trawl/blob/main/LICENSE). Not affiliated with or endorsed by Tavily — Trawl exists so you can *develop against* their API shape without spending credits, and hopefully become a happier paying customer in production.
