Metadata-Version: 2.4
Name: web-scout-ai
Version: 1.9.0
Summary: Agentic web research tool — smarter than search, faster than deep research. Search, scrape, and synthesize web content using LLMs.
License: MIT
License-File: LICENSE
Keywords: web-research,ai-agent,web-scraping,llm,search,rag,crawling,document-extraction,pdf-extraction,playwright,docling,synthesis,openai-agents,serper,duckduckgo,litellm,grounded-research
Author: Riccardo Soldan
Requires-Python: >=3.13,<3.14
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pydantic :: 2
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Dist: boto3 (>=1.43.61,<2.0.0)
Requires-Dist: docling (>=2.0)
Requires-Dist: docling-core (>=2.0)
Requires-Dist: httpx (>=0.24)
Requires-Dist: litellm (>=1.30)
Requires-Dist: markdownify (>=0.11)
Requires-Dist: openai-agents (>=0.13)
Requires-Dist: pdf-extractor-ai (>=0.2.1)
Requires-Dist: prefect (>=3.4,<4)
Requires-Dist: pydantic (>=2.0)
Requires-Dist: pydantic-settings (>=2.0)
Requires-Dist: scrapling[fetchers] (>=0.4.15)
Requires-Dist: typesafe-sdk (>=0.7.2,<0.8.0)
Project-URL: Repository, https://github.com/RSO9192/web-scout-ai
Description-Content-Type: text/markdown

# web-scout-ai

![web-scout-ai logo](assets/web-scout-logo.svg)

[![PyPI Version](https://img.shields.io/pypi/v/web-scout-ai)](https://pypi.org/project/web-scout-ai/)
[![Downloads](https://static.pepy.tech/badge/web-scout-ai/month)](https://pepy.tech/project/web-scout-ai)
[![Python Versions](https://img.shields.io/pypi/pyversions/web-scout-ai)](https://pypi.org/project/web-scout-ai/)
[![License](https://img.shields.io/github/license/RSO9192/web-scout-ai)](LICENSE)

**Web research that actually reads the sources.**

`web-scout-ai` is an agentic Python research pipeline that searches the web, opens and extracts the sources it finds, follows up when evidence is incomplete, and returns a cited answer together with a full audit trail of what it could — and could not — read.

Give it a question or a URL:

```python
from web_scout import run_web_research

result = await run_web_research(
    "Indonesia biodiversity hotspots current status and recent trend"
)

print(result.synthesis)
print(f"Read {len(result.scraped)} sources across {len(result.queries)} searches")
```

Instead of treating search snippets as evidence, `web-scout-ai` tries to **read the underlying source itself** before allowing it to support the final answer.

<details open>
<summary><strong>Example output</strong> (excerpt)</summary>

Query: `Indonesia biodiversity hotspots current status and recent trend`

> Indonesia remains one of the world's most important centres of biodiversity,
> but the available evidence indicates a negative overall status and trend. The
> UNEP country fiche describes Indonesia as one of 17 megadiverse countries,
> containing two globally recognised biodiversity hotspots; it reports
> approximately 31,750 plant species, 732 mammals, 1,711 birds, 750 reptiles,
> 403 amphibians and 1,236 freshwater fish.
> [Biodiversity / Indonesia | Interactive Country Fiches](https://dicf.unepgrid.ch/indonesia/biodiversity)
>
> A 2024 review reports that, according to Ministry of Environment and Forestry
> data for 2020, Indonesia retained 86.9 million hectares of forest, including
> 45.3 million hectares of primary forest. In Sundaland, 70.6% of the region
> had an intense human-footprint score (2009 dataset); among 308 lowland-forest
> bird species assessed, 273, or 88.6%, experienced a reduction in suitable
> range between 2000 and 2015.
> [Forest fragmentation in Indonesia](https://www.sciencedirect.com/science/article/pii/S2351989424001227),
> [Severe human pressures in the Sundaland biodiversity hotspot](https://conbio.onlinelibrary.wiley.com/doi/10.1111/csp2.169)
>
> The most recent forest-loss signal is mixed but concerning. WRI/University of
> Maryland analysis reports that Indonesia's tropical primary-forest loss
> increased 14% from 2024 to 2025, although it remained below the high levels
> of the mid-2010s. A global land-use analysis covering 1995–2022 found that
> Indonesia accounted for 22% of global biodiversity losses attributed to
> land-use change, the largest national share in the study.
> [Tropical rainforest loss slowed in 2025](https://gfr.wri.org/latest-analysis-deforestation-trends),
> [Biodiversity impacts of recent land-use change](https://www.nature.com/articles/s41893-024-01433-4)
>
> Comparisons should be made cautiously because the sources use different
> indicators and years: human-footprint data end in 2009, bird-range data in
> 2015, several forest assessments in 2020–2022, and primary-forest-loss data
> in 2025.

This is a fraction of what that call returned. The full synthesis was 10,055 characters; `result.scraped` also held extracted evidence from all 9 sources (~34,000 characters)—including a ScienceDirect review, a Conservation Biology paper, a Nature paper, a CEPF ecosystem profile, an interactive UNEP country fiche, and WRI's 2025 forest-loss analysis—plus an audit trail of the 3 searches, 1 irrelevant fetch, and 13 snippet-only hits that were not used as citations. The run took 188.7 seconds. Results and timing vary with the query, sources, model, and network.

</details>

## What it actually does

`web-scout-ai` is an async Python research pipeline, not a search-result wrapper. It searches, opens the selected sources, routes each response to the right extractor, evaluates whether the collected evidence is sufficient, and synthesizes only from sources it successfully read.

| Input encountered | What the pipeline does |
| --- | --- |
| Static HTML | Uses a fast HTTP fetch and converts the page to clean Markdown |
| JS-heavy pages and SPA shells | Escalates to a stealth Chromium browser and can interact with page controls |
| Cloudflare or similar bot challenges | Retries with Scrapling's stealth browser and Cloudflare challenge handling |
| PDF, DOCX, PPTX, XLSX | Downloads and converts the document with Docling |
| Scanned PDFs, charts, maps, images | Uses the configured vision model when text extraction is insufficient |
| JSON endpoints | Converts structured payloads into readable evidence |
| List, index, and database pages | Ranks detail links, follows them, and can take one pagination hop |
| Thin or incomplete evidence | Scrapes promising backlog URLs or generates targeted follow-up searches |

The browser fallback can pass many JS-gated and Cloudflare-protected pages, but it is not a guarantee: sites can still block automation. Those URLs are reported in `result.bot_detected` instead of disappearing or being presented as evidence. Use the package only where you have permission and in accordance with the source site's terms.

## Quick start

### 1. Install

```bash
pip install web-scout-ai
web-scout-setup
```

`web-scout-setup` installs the Patchright-managed Chromium browser and its system dependencies. It may request `sudo` for OS-level browser libraries.

### 2. Configure keys

The default research models run on Amazon Bedrock Mantle. Vision fallback still uses Gemini, and open-web discovery uses Serper:

```bash
export AWS_BEARER_TOKEN_BEDROCK="your-bedrock-api-key"
export GEMINI_API_KEY="your-gemini-api-key"
export SERPER_API_KEY="your-serper-api-key"
```

`BEDROCK_MANTLE_API_KEY` is accepted as an alias for the bearer token. OpenAI models on Mantle, including GPT-5.6 and GPT-6 Luna, use LiteLLM's route in `us-east-1`.

To use the Exa search backend instead (`search_backend="exa"`):

```bash
export EXA_API_KEY="your-exa-api-key"
```

Direct-URL mode does not use a search backend, so it only needs the API key for your configured model provider.

### 3. Run research

```python
import asyncio

from web_scout import run_web_research


async def main():
    result = await run_web_research(
        query="What are the main threats to coral reefs worldwide?",
        cache=True,
    )

    print(result.synthesis)

    print("\nSources read:")
    for source in result.scraped:
        print(f"- {source.title or source.url}: {source.url}")

    if result.bot_detected or result.scrape_failed:
        print(
            f"\nCould not read "
            f"{len(result.bot_detected) + len(result.scrape_failed)} source(s)"
        )


asyncio.run(main())
```

## Three ways to use it

### Open-web research

Generate several searches, read the strongest results in parallel, evaluate coverage, and search again if important evidence is missing.

```python
result = await run_web_research(
    query="What is driving the global adaptation finance gap?",
)
```

### Domain-restricted research

Keep discovery and hub deepening focused on authoritative domains.

```python
result = await run_web_research(
    query="Latest evidence on sea-level rise",
    include_domains=["ipcc.ch", "nasa.gov"],
)
```

### Direct URL extraction

Skip search and start from a page, document, API endpoint, image, or database listing.

```python
result = await run_web_research(
    query="Extract the recommended adaptation measures and supporting evidence",
    direct_url="https://example.org/report.pdf",
)
```

For a document, the pipeline reads that document without wandering into site navigation. For a list or database page, it can rank and follow relevant records and their linked primary documents.

## The return value is an audit trail

`run_web_research()` returns a typed `WebResearchResult`:

```python
class WebResearchResult(BaseModel):
    synthesis: str
    scraped: list[UrlEntry]
    scrape_failed: list[UrlEntry]
    blocked_by_policy: list[UrlEntry]
    source_http_error: list[UrlEntry]
    scraped_irrelevant: list[UrlEntry]
    bot_detected: list[UrlEntry]
    snippet_only: list[UrlEntry]
    queries: list[SearchQuery]
```

| Field | Meaning |
| --- | --- |
| `synthesis` | Final answer with inline Markdown citations |
| `scraped` | Sources successfully read and extracted; `content` contains the query-relevant evidence |
| `scrape_failed` | Extraction attempts that failed for an unclassified reason |
| `blocked_by_policy` | Sources skipped because their domain is in `exclude_domains` |
| `source_http_error` | Source-side HTTP or network failures |
| `scraped_irrelevant` | Pages fetched successfully but not useful for the query |
| `bot_detected` | Sources that still returned a bot-protection wall |
| `snippet_only` | Search results discovered but not opened; snippets are never valid citation targets |
| `queries` | Every search query executed, its result count, and domain restrictions |

`UrlEntry` contains `url`, `title`, and `content`. `SearchQuery` contains `query`, `num_results_returned`, and `domains_restricted`.

The synthesizer is instructed to use scraped evidence only. A deterministic final check rejects citations to invented or snippet-only URLs and retries the synthesis with feedback.

## Research depth

```python
# Faster default
await run_web_research(query="...", research_depth="standard")

# More searches and sources, with a stricter coverage threshold
await run_web_research(query="...", research_depth="deep")
```

| Budget | Standard | Deep |
| --- | ---: | ---: |
| Maximum search iterations | 2 | 3 |
| Initial search queries | 3 | 5 |
| Follow-up search queries | 2 | 4 |
| URLs selected in the first round | 6 | 12 |
| URLs selected in a follow-up round | 4 | 8 |
| Hub/detail-page cap | 10 | 15 |

The coverage evaluator can stop early when the evidence already answers the question. You can add your own acceptance conditions with `coverage_criteria`:

```python
result = await run_web_research(
    query="Compare national methane policies",
    coverage_criteria="Include at least one primary government source per country.",
)
```

## Custom extractor guidance

Use `extractor_guidance` to refine what each per-source extractor keeps and how it
organizes evidence. The guidance applies only to extraction; it does not alter search
query generation, coverage evaluation, or final synthesis.

```python
result = await run_web_research(
    query="Tunisia agricultural trends",
    extractor_guidance="""
Country of interest: Tunisia

- Keep findings explicitly about Tunisia.
- Also keep regional findings whose stated scope includes Tunisia, even when the
  source does not name Tunisia; label them with the source's scope, such as
  "North Africa:".
- Discard purely global findings and findings about geographies that do not
  include Tunisia.
""",
)
```

Guidance augments rather than replaces WebScout's extractor instructions. The base
contract always takes precedence for source grounding, tool constraints, output
fields, page-type handling, and the exact no-evidence sentinel.

## Configuration

The defaults are Bedrock Mantle GPT-6 Luna for research and extraction, TypeSafe Jev for follow-up link selection, and `gemini/gemini-3.7-flash` for vision. Model IDs follow [LiteLLM provider naming](https://docs.litellm.ai/docs/providers), so each stage can use a different provider or model. Set `WEB_SCOUT_FOLLOWUP_BACKEND=gpt` to use the Agents SDK model in `models["followup_selector"]` instead of Jev. Jev reads `TYPESAFE_API_KEY`.

```python
models = {
    # Used as the fallback for query generation, coverage, and synthesis
    "web_researcher": "openai/gpt-4o-mini",

    # Reads and extracts individual sources
    "content_extractor": "gemini/gemini-2.0-flash",

    # Optional stage-specific choices
    "query_generator": "openai/gpt-4o-mini",
    "coverage_evaluator": "openai/gpt-4o-mini",
    "synthesiser": "openai/gpt-4o-mini",
    "followup_selector": "openai/gpt-4o-mini",  # used when WEB_SCOUT_FOLLOWUP_BACKEND=gpt
    "vision_fallback": "gemini/gemini-2.0-flash",
}

result = await run_web_research(query="...", models=models)
# Or keep Luna for link selection:
# Set WEB_SCOUT_FOLLOWUP_BACKEND=gpt in the environment, then pass models as usual.
```

Provider credentials are read from their standard environment variables, such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `TYPESAFE_API_KEY`, or AWS credentials for Bedrock.

### Public API

Fetch one URL and get a `FetchResult`. This uses the same Scrapling fetcher as the research pipeline (fast HTTP, then a stealth browser when the page looks blocked or JavaScript-rendered):

```python
from web_scout import PDF_MAGIC_BYTES, fetch

result = await fetch("https://example.org/report.pdf")
page = await fetch("https://example.org/article", wait_for="article")
```

`result.body` is the raw response bytes for binary payloads such as PDFs and images. HTML and other text responses leave `body` as `None` and put the decoded page in `result.html_content`. Failures do not raise: `result.error` is set when the URL is rejected or the fetch fails, and `result.status` is outside the 2xx range for HTTP errors. Pass `exclude_domains` to skip hosts before any network call.

`PDF_MAGIC_BYTES` is `b"%PDF"`, the four-byte header every PDF file starts with. Check the body against it when you need to know whether the fetch returned a PDF. That is more reliable than `result.content_type`, which download URLs often report as `application/octet-stream`:

```python
is_pdf = bool(result.body and result.body.startswith(PDF_MAGIC_BYTES))
if is_pdf:
    pdf_bytes = result.body
```

`fetch_pdf` is a thin wrapper around `fetch` that does this check and returns the PDF bytes. It raises `RuntimeError` when the fetch fails or the body does not start with `PDF_MAGIC_BYTES`:

```python
from web_scout import fetch_pdf

pdf_bytes = await fetch_pdf("https://openknowledge.fao.org/bitstreams/c1caede2-ea98-46b0-b663-4cae429e05d3/download")
```

```python
result = await run_web_research(
    query="latest IPCC findings on sea-level rise",
    models=None,                       # optional; GPT-6 Luna defaults, Gemini vision
    search_backend="serper",          # "serper" (Google via serper.dev) or "exa" (exa.ai)
    research_depth="standard",        # "standard", "deep", or a custom dict
    include_domains=["ipcc.ch"],       # optional discovery restriction
    direct_url=None,                   # optional; skips search when set
    domain_expertise="climate science",  # optional prompt context
    exclude_domains=None,              # optional blocklist; nothing is blocked
                                       # by default. See "Domain policy" below.
    max_pdf_pages=50,                  # pages converted from each PDF
    max_content_chars=30_000,          # characters passed to the extractor per source
    cache=False,                       # accepted; URL and PDF caches stay on
    cache_storage=None,               # Prefect block, e.g. a GCS bucket
    refresh_pdf_cache=False,          # recompute cached PDF parses
    refresh_url_cache=False,          # recompute cached URL fetches
    coverage_criteria=None,            # extra evidence requirements
    extractor_guidance=None,           # optional per-source extraction guidance
)
```

### Domain policy

Nothing is blocked by default: `exclude_domains=None` means every domain is eligible for search and scraping. To block domains, pass a list explicitly. A curated list of platforms that waste the scrape budget (social/video sites with no scrapeable text, bot-blocked search engines, hard-paywalled publishers) is exported as `RECOMMENDED_EXCLUDE_DOMAINS`:

```python
from web_scout import run_web_research, RECOMMENDED_EXCLUDE_DOMAINS

# Opt in to the curated list
result = await run_web_research(
    query="...",
    exclude_domains=RECOMMENDED_EXCLUDE_DOMAINS,
)

# Curated list plus your own additions
result = await run_web_research(
    query="...",
    exclude_domains=[*RECOMMENDED_EXCLUDE_DOMAINS, "statista.com"],
)
```

Exclusions are applied natively by search backends that support them (Exa's `excludeDomains`); backends without native support (Serper) rely on the post-search URL filter, which always runs. Contradictions fail fast: a domain appearing in both `include_domains` and `exclude_domains`, or a `direct_url` whose host is excluded, raises `ValueError`.

### Source caching

URL fetches and PDF parses are cached with Prefect. A second request for the same URL reuses the stored page or document. A second request for the same PDF bytes reuses the parse even when the URL is different. Query-specific extraction and synthesis still run each time.

By default the cache is on disk at `~/.cache/web-scout` and survives a new process. Pass `cache_storage` to store both caches in a caller-supplied Prefect block, such as a GCS bucket:

```python
from prefect_gcp.cloud_storage import GcsBucket

result = await run_web_research(
    query="...",
    cache_storage=GcsBucket(bucket="my-cache"),
    refresh_pdf_cache=False,
    refresh_url_cache=False,
)
```

`refresh_pdf_cache=True` and `refresh_url_cache=True` recompute those entries. `cache=True` is still accepted and does not turn the cache off. Failed scrapes, final answers, query-specific summaries, and click-driven browser sessions are not cached. Concurrent workers wait rather than fetching or parsing the same URL or PDF at the same time.

## Pipeline, in one view

```text
question
   │
   ├─ generate targeted searches ─ search in parallel ─ select diverse URLs
   │                                                     │
   │                                                     ▼
   │      static HTML ────────┐                    fetch in parallel
   │      JS / bot challenge ─┤                          │
   │      documents ──────────┼─ route + extract ◀───────┘
   │      JSON / images ──────┘          │
   │                                    ▼
   └─ follow-up search ◀── evaluate evidence coverage
                                        │ sufficient
                                        ▼
                              grounded synthesis
                                        │
                                        ▼
                               citation validation
```

## Where it fits

Use `web-scout-ai` when your application needs the contents of real pages and documents—not only search snippets—and you want the source successes and failures returned as structured data.

It is intentionally a bounded research component. If you only need search links, use a search API directly. If you need an open-ended autonomous research process with human checkpoints, put this package inside a broader agent workflow.

## Requirements

- Python 3.13
- An API key for the configured LLM provider
- A Serper or Exa API key for search mode
- Chromium setup for rendered pages, interactive sites, and browser fallbacks

## Classification and resource reuse

Semantic classification defaults to TypeSafe Jev and requires `TYPESAFE_API_KEY`. Set `WEB_SCOUT_CLASSIFICATION_BACKEND=gpt` to use the existing GPT classifiers instead. Query generation, factual extraction, visual descriptions and synthesis continue to use your configured generative models. Feature-specific switches and the global override are described below.

CPU PDF pipelines initialize on the first document and are reused. Browser and HTTP resources are reused on the same event loop; after requests finish, call `await web_scout.scraping.close_resources()` at application shutdown. See the [performance report](docs/performance.md) for measurements, limits, benchmark commands and compatibility details.

## Contributing

The main extension point is [`SearchBackend`](src/web_scout/search_backends.py). New backends should implement the async `search()` contract and return normalized results (title, url, snippet).

Bug reports and focused pull requests are welcome at [github.com/RSO9192/web-scout-ai](https://github.com/RSO9192/web-scout-ai).

## License

[MIT](LICENSE)


### Jev / GPT selection

Set `DISABLE_JEV=true` to use the retained GPT implementations for every Jev-backed feature. It overrides all feature settings and does not require `TYPESAFE_API_KEY`. Unset or `DISABLE_JEV=false` preserves Jev defaults. Values are case-insensitive and surrounding whitespace is ignored.

| Environment variable | Feature | Values |
| --- | --- | --- |
| `WEB_SCOUT_CRAWLER_BACKEND` | Crawler link classification | `jev`, `gpt` |
| `WEB_SCOUT_COVERAGE_BACKEND` | Search coverage evaluation | `jev`, `gpt` |
| `WEB_SCOUT_FOLLOWUP_BACKEND` | Follow-up link selection | `jev`, `gpt` |
| `WEB_SCOUT_PDF_CLAIMS_BACKEND` | PDF claim verification when enabled | `jev`, `gpt` |

Each feature defaults to `jev`. The existing `WEB_SCOUT_CLASSIFICATION_BACKEND` remains a shared default for features without a specific setting. Invalid backend values raise an error; missing Jev credentials remain an error when an active feature selects Jev. GPT paths use the existing configured models and provider credentials; disabling Jev does not change model IDs or switch Bedrock models to the OpenAI endpoint.

The `followup_backend` argument has been removed from `run_web_research()`. Replace `followup_backend="luna"` with `WEB_SCOUT_FOLLOWUP_BACKEND=gpt`, or `followup_backend="jev"` with `WEB_SCOUT_FOLLOWUP_BACKEND=jev`.


### Orchestrator link selection

`Orchestrator` uses `DefaultCrawler`, which selects links from already-parsed content using the same Jev/GPT selectors as the research pipeline. Select the backend with `WEB_SCOUT_CRAWLER_BACKEND=jev|gpt`; `DISABLE_JEV=true` forces GPT. `WEB_SCOUT_FOLLOWUP_BACKEND` independently controls follow-ups in `run_web_research()`.

`Crawl4AICrawler` remains an alias for `DefaultCrawler` to preserve existing imports. `llm_config=None` still requests heuristic-only selection. Existing config objects with `provider`, `api_token` and `base_url` are accepted for custom GPT connections; third-party extraction-strategy options are no longer used. No Crawl4AI package, browser or setup is required.

