Metadata-Version: 2.4
Name: deep-search-agent
Version: 0.4.1
Summary: Deep web search agent library built on LangChain deepagents: orchestrator + specialized subagents + iterative rubric-graded refinement.
Project-URL: Homepage, https://github.com/giurlanda/deep-search-agent
Project-URL: Documentation, https://giurlanda.github.io/deep-search-agent/
Project-URL: Repository, https://github.com/giurlanda/deep-search-agent
Project-URL: Issues, https://github.com/giurlanda/deep-search-agent/issues
Author-email: Hawk <francesco.giurlanda@gmail.com>
License: MIT
License-File: LICENSE
Keywords: agent,deep-search,deepagents,langchain,llm,research,search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: deepagents>=0.6.5
Requires-Dist: httpx>=0.28.1
Requires-Dist: langchain-core<2.0,>=1.0
Requires-Dist: langchain<2.0,>=1.0
Requires-Dist: pypdf>=6.14.2
Requires-Dist: trafilatura>=2.1.0
Provides-Extra: benchmark
Requires-Dist: langchain-openai<2.0,>=1.0; extra == 'benchmark'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'docs'
Provides-Extra: js-render
Requires-Dist: playwright>=1.49; extra == 'js-render'
Description-Content-Type: text/markdown

# deep-search-agent

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Version](https://img.shields.io/github/v/tag/giurlanda/deep-search-agent?sort=semver&label=version)](https://github.com/giurlanda/deep-search-agent/tags)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)

A Python library for deep internet searches (deep search), similar to the deep
research features of ChatGPT and Claude. Built on top of LangChain's
[deepagents](https://docs.langchain.com/oss/python/deepagents).

📖 **Documentation:** <https://giurlanda.github.io/deep-search-agent/>

## Architecture

The `create_deep_search_agent` factory returns a deep agent configured with the
**orchestrator + specialized sub-agents + evaluation loop** pattern:

| Component | Implementation |
|---|---|
| Orchestrator | Main agent (`create_deep_agent`): decomposes the query with `write_todos`, delegates, synthesizes with citations |
| `perspective-agent` | Explores the topic from 3-6 distinct angles (analysis axes, stakeholder viewpoints, dimensions of the problem) before decomposition, saved to `/research/perspectives.md`; enabled by default, toggle with `enable_perspectives=False` |
| `search-agent` | Web search via SearxNG (+ optional additional search tools), reformulates queries, saves results with their source |
| `fetch-agent` | Downloads and extracts content from URLs: clean HTML with `trafilatura`, PDFs read with `pypdf`, User-Agent from real browsers |
| `fact-check-agent` | Verifies claims against multiple independent sources (has both search and fetch) |
| Shared memory | deepagents virtual filesystem: each sub-agent writes `/findings/<source-slug>.md` with URL, date, and claims |
| Shared source index | `/findings/_sources.md`: one line per URL (`saved` / `failed` / `discarded`) that every agent consults and appends to, so searches and fetches are not duplicated across cycles |
| Evaluator/critic | `RubricMiddleware` (beta): an LLM grader evaluates the answer against a rubric and re-runs the orchestrator up to `max_research_cycles` cycles |

Each sub-agent runs with an isolated context: raw page content does not pollute
the orchestrator's memory; only the synthetic reports and the files in
`findings/` bubble up.

## Installation

```bash
uv sync            # from the repository
# or, as a dependency:
uv add deep-search-agent
```

Requires Python ≥ 3.12. The search tool needs a reachable
[SearxNG](https://docs.searxng.org/) instance with the JSON format enabled
(default: `http://localhost:8888`).

## Quickstart

```python
from deep_search_agent import create_deep_search_agent

agent = create_deep_search_agent(
    model="anthropic:claude-sonnet-4-6",
    searxng_base_url="http://localhost:8888",
    max_research_cycles=3,
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "State of the art in quantum error correction in 2026?"}]},
    config={"configurable": {"thread_id": "research-1"}},
)
print(result["messages"][-1].content)
```

The default evaluation rubric (`DEEP_SEARCH_RUBRIC`) is injected automatically:
the refinement loop works with no configuration. For an ad-hoc rubric, pass it
in the invoke state (`{"rubric": "- ..."}`) or to the factory
(`rubric="- ..."`).

## Factory parameters

Deep-search-specific parameters:

| Parameter | Default | Description |
|---|---|---|
| `model` | — (required) | Orchestrator model; inherited by sub-agents and the rubric grader |
| `max_research_cycles` | `3` | Maximum refinement cycles of the evaluator loop (and budget cited in the orchestrator prompt) |
| `max_query_variants` | `3` | Number of parallel query variants the search agent issues per sub-question (synonyms, broader/narrower terms, English variants) to widen recall |
| `max_search_results_per_query` | `5` | Maximum results per search query |
| `max_urls_to_scrape_per_cycle` | `3` | Maximum URLs to fetch per research cycle |
| `searxng_base_url` | `http://localhost:8888` | URL of the SearxNG instance |
| `searxng_engines` | `None` | List of SearxNG engines to restrict the search to |
| `searxng_rate_limit` | `None` | Minimum seconds between SearxNG requests (thread-safe, shared across concurrent searches); `None` disables rate limiting |
| `searxng_budget` | `None` | Maximum SearxNG searches per research cycle; when exhausted the tool returns an `ERROR:` telling the model no budget is left. `None` means unlimited |
| `request_timeout` | `15.0` | HTTP timeout (s) for search and fetch |
| `max_content_chars_per_page` | `20000` | Truncation of extracted content per page |
| `enable_js_render_fallback` | `False` | Re-fetch pages whose static HTML yields no content through a headless Chromium, recovering JavaScript-only pages and bot walls. Requires the `js-render` extra plus `playwright install chromium` |
| `js_render_timeout` | `30.0` | Seconds the headless renderer waits for a page to settle; ignored unless the fallback is enabled |
| `search_tools` | `None` | Additional search tools for search-agent, fact-check-agent, and perspective-agent (e.g. Tavily, RAG retrieval) |
| `enable_perspectives` | `True` | Adds `perspective-agent` and instructs the orchestrator to delegate to it before decomposing the query. Set `False` for simple queries where a single-axis decomposition is sufficient |
| `rubric` | `DEEP_SEARCH_RUBRIC` | Custom evaluation rubric |
| `auto_rubric` | `True` | Auto-inject the rubric into the state on every invoke |
| `on_evaluation` | `None` | Callback invoked with each `RubricEvaluation` after the grader scores a cycle (e.g. to log verdicts); exceptions are logged and suppressed |
| `subagents_middleware` | `()` | Extra middleware injected into each built-in sub-agent (perspective-agent, search-agent, fetch-agent, fact-check-agent) |
| `subagents` | `None` | Extra sub-agents, added to the built-in ones |
| `backend` | `StateBackend()` | Filesystem backend shared by the orchestrator and every sub-agent |

All other keyword arguments (`tools`, `checkpointer`, `store`, `skills`,
`interrupt_on`, ...) are passed through unchanged to `create_deep_agent`.

## Extensions

### Adding a retrieval agent (RAG) over the internal knowledge base

```python
rag_agent = {
    "name": "rag-agent",
    "description": "Retrieval over the internal company knowledge base",
    "system_prompt": "Search the vector store and save results to findings/.",
    "tools": [my_vector_store_tool],
}

agent = create_deep_search_agent(
    model="anthropic:claude-sonnet-4-6",
    subagents=[rag_agent],
)
```

### Adding search engines

```python
from langchain_tavily import TavilySearch

agent = create_deep_search_agent(
    model="anthropic:claude-sonnet-4-6",
    search_tools=[TavilySearch(max_results=5)],
)
```

### Persistent backend for findings

```python
from deepagents.backends import FilesystemBackend

agent = create_deep_search_agent(
    model="anthropic:claude-sonnet-4-6",
    backend=FilesystemBackend(root_dir="./research", virtual_mode=True),
)
```

## Testing

```bash
uv run pytest
```

The unit tests require neither network nor API keys: HTTP and LLM are simulated.

## Publishing

Releases are published to [PyPI](https://pypi.org/project/deep-search-agent/) by
the [`publish` workflow](.github/workflows/publish.yml), triggered on pushing a
`v*` tag. It uses **PyPI Trusted Publishing** (OIDC), so no API token is stored.

One-time setup on PyPI (Account → Publishing → *Add a pending publisher*):

| Field | Value |
|---|---|
| PyPI project name | `deep-search-agent` |
| Owner | `giurlanda` |
| Repository name | `deep-search-agent` |
| Workflow name | `publish.yml` |
| Environment name | `pypi` |

To cut a release: bump `__version__` in
[`src/deep_search_agent/__init__.py`](src/deep_search_agent/__init__.py), update
the `CHANGELOG`, then push a matching tag (e.g. `git tag v0.4.1 && git push
origin v0.4.1`). Build locally with `uv build` and validate with
`uvx twine check dist/*`.

## Package structure

```
src/deep_search_agent/
├── factory.py       # create_deep_search_agent
├── prompts.py       # orchestrator/sub-agent prompts + default rubric
├── middleware.py    # DefaultRubricMiddleware (rubric auto-injection)
├── subagents.py     # perspective/search/fetch/fact-check agent definitions
└── tools/
    ├── search.py    # SearxNG tool
    └── fetch.py     # URL fetch tool (trafilatura + pypdf)
```
