Metadata-Version: 2.4
Name: searchnow
Version: 0.2.3
Summary: Open-source web search and page extraction (text, images, HTML) for LLMs and AI agents, powered by self-hosted SearXNG.
Author: Sahil Jain
License: MIT
Project-URL: Homepage, https://github.com/sahiljain/searchnow
Project-URL: Repository, https://github.com/sahiljain/searchnow
Project-URL: Issues, https://github.com/sahiljain/searchnow/issues
Project-URL: Changelog, https://github.com/sahiljain/searchnow/blob/main/CHANGELOG.md
Keywords: search,searxng,llm,agents,rag,web-scraping,trafilatura,tavily-alternative
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx[http2]<1.0,>=0.27
Requires-Dist: trafilatura>=1.12
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: respx>=0.21; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.1; extra == "dev"
Dynamic: license-file

# SearchNow

**Open-source web search and page extraction for LLMs and AI agents.**

One class. Give it a URL and get back the **content, the images and the raw
HTML**. Give it a query and get back search results — from your own
[SearXNG](https://docs.searxng.org/) instance, so nothing leaves your
infrastructure and there is no per-query bill.

A self-hosted replacement for Tavily / Brave Search / SerpAPI.

```python
from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://en.wikipedia.org/wiki/Python_(programming_language)")

page.text          # readable markdown, nav and cookie banners stripped
page.html          # the raw HTML source
page.images        # [Image(url=..., alt=..., width=..., height=...), ...]
page.main_image    # the page's own preview image
page.title, page.author, page.description, page.published_date, page.sitename
```

---

## Install

```bash
pip install git+https://github.com/sahiljain/searchnow
```

**That's the whole setup.** You need Docker installed and running; SearchNow
handles the rest. The first time you `search()`, if no SearXNG is answering, it
starts one for you:

```python
from searchnow import FetchResult

fr = FetchResult()
fr.search_sync("nvidia nemo framework")
# WARNING no SearXNG instance answering on port 8080 -- setting one up in
#         ~/.searchnow/searxng (the first run pulls a ~500 MB Docker image...)
# WARNING SearXNG is up. Stop it with: cd ~/.searchnow/searxng && docker compose down
```

Roughly 2-4 minutes the first time (Docker image pull), a couple of seconds
after that. If you delete the folder or stop the containers, the next `search()`
rebuilds it.

### How it decides

| | |
|---|---|
| Something already answering on `base_url` | does nothing |
| Nothing there, `base_url` is local | runs the setup |
| Nothing there, `base_url` is **remote** | raises — it will not "fix" someone else's server by starting a local container |
| Docker missing or not running | raises, and says exactly what to install or start |
| You called `fetch()`, not `search()` | does nothing — fetching a URL needs no SearXNG |

The script (`setup.sh`) **ships inside the package**, so nothing is downloaded
at runtime. You can also run it yourself:

```bash
script=$(python -c "import searchnow; print(searchnow.bootstrap.script_path())")
bash "$script" --dir searxng --port 8080
#   --port 9090     a different port
#   --dir mysearx   a different folder (its own Compose project, no collisions)
#   --force         overwrite an existing config
#   --no-start      write the files, start it yourself
```

### Turning it off

```python
fr = FetchResult("https://searx.mycompany.internal", auto_setup=False)
```

or export `SEARCHNOW_AUTO_SETUP=0`. (An env var can only turn it *off* — an
environment should not be able to opt you into starting containers.) You can
also control *when* the slow first setup happens, rather than having it land
mid-request:

```python
fr = FetchResult()
fr.ensure_searxng_sync()      # at startup; await fr.ensure_searxng() in async code
```

> A **stock** `searxng/searxng` image will not work: it ships
> `search.formats: [html]`, so `format=json` answers 403, and its bot limiter
> blocks every non-browser client. `setup.sh` fixes both, plus enables a set of
> engines that actually respond. Details in [docs/searxng.md](docs/searxng.md).

---

## Usage

### Fetch a URL

```python
from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://example.com/article")

if page:                        # a Page is truthy when ok
    print(page.title)
    print(page.text)
    for image in page.images:
        print(image.url, image.alt, image.width, image.height)
else:
    print("failed:", page.error)
```

A failed fetch is a `Page` with `ok=False`, **not an exception** — one dead link
must never take down a batch of ten. Pass `raise_on_error=True` if you want the
opposite.

> `fetch()` takes a **URL**, `search()` takes a **query**. The 0.0.x names still
> work and map the same way: `fetch_url` → `fetch`, `fetch_data` → `search`.
> Passing a query to `fetch()` tells you so rather than failing at DNS:
>
> ```python
> fr.fetch_sync("nemo").error
> # "this is not a URL. fetch() downloads one page ... To look a topic up,
> #  use search() instead -- e.g. fr.search_sync('nemo')"
> ```

#### Sites that block scripts

Some sites (Flipkart, and anything behind Cloudflare's challenge, DataDome,
PerimeterX…) serve a JavaScript CAPTCHA instead of the page. **No plain HTTP
client can get past that** — not this library, not `curl`, not `requests`.
SearchNow names the wall instead of leaving you with a bare `403`:

```python
page = fr.fetch_sync("https://www.flipkart.com/some-product/p/itm123")

page.ok       # False
page.blocked  # True  <- an anti-bot challenge, not a bug in your code
page.error
# 'HTTP 403 -- blocked by an anti-bot challenge (Google reCAPTCHA). The site
#  only serves this page to a browser that can run its JavaScript challenge...'
```

`blocked` means *retrying will never work*, so branch on it rather than on
`status_code`:

```python
pages = fr.fetch_many_sync(urls)
usable = [p for p in pages if p.ok]
needs_browser = [p.url for p in pages if p.blocked]   # hand these to Playwright
```

### Search

```python
response = fr.search_sync("nvidia nemo framework", max_results=5)

for result in response:
    print(result.score, result.title, result.url)
    print(result.content)               # the engine's snippet

print(response.urls)
```

### Search *and* read every result — one call

```python
response = fr.search_sync(
    "how does speculative decoding work",
    max_results=5,
    include_content=True,       # downloads all 5 pages concurrently
    max_chars=4000,
)

for result in response:
    print(result.url, len(result.page.text), len(result.page.images))

print(response.to_context())    # numbered, cited markdown for a prompt
```

### Many URLs at once

```python
pages = fr.fetch_many_sync([
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/c",
])
# Concurrent, input order preserved, failures included as ok=False pages.
```

### Async

Every method has an async twin. Same object, same arguments — drop the `_sync`:

| sync | async |
|---|---|
| `fr.search_sync(q)` | `await fr.search(q)` |
| `fr.fetch_sync(url)` | `await fr.fetch(url)` |
| `fr.fetch_many_sync(urls)` | `await fr.fetch_many(urls)` |

```python
import asyncio
from searchnow import FetchResult

async def main():
    async with FetchResult() as fr:
        response = await fr.search("retrieval augmented generation", max_results=5)
        pages = await fr.fetch_many(response.urls)
        print(pages[0].text)

asyncio.run(main())
```

The sync methods run on a private event loop in a background thread, so they
also work **inside a Jupyter notebook**, where `asyncio.run` would raise "this
event loop is already running". You can mix sync and async calls on one
instance.

### As an LLM tool

```python
from searchnow import FetchResult

fr = FetchResult(cache_ttl=300)      # one instance for the process

async def web_search(query: str, max_results: int = 5) -> str:
    """Search the web and return sourced, readable content."""
    response = await fr.search(query, max_results=max_results, include_content=True)
    return response.to_context(max_chars_per_result=3000)
```

Keep **one** `FetchResult` alive. Building a new one per call throws away the
connection pool and the cache, which is the usual reason this feels slow.

---

## Parameters

All optional, all on the constructor, all overridable per call where it makes
sense.

```python
fr = FetchResult(
    "http://localhost:8080",     # or $SEARXNG_URL
    max_results=10,
    engines="bing,mojeek",
    cache_ttl=300,
    max_concurrency=16,
)
```

| Parameter | Default | What it does |
|---|---|---|
| `base_url` | `$SEARXNG_URL` or `http://localhost:8080` | Your SearXNG instance |
| `max_results` | `5` | Default result count |
| `engines` | *(instance default)* | `"bing,mojeek"` or `["bing", "mojeek"]` |
| `categories` | `"general"` | `general`, `news`, `science`, `it`, `images`, … |
| `language` | `"en"` | Result language |
| `safesearch` | `0` | 0 off, 1 moderate, 2 strict |
| `time_range` | `None` | `day` / `week` / `month` / `year` |
| `search_timeout` | `30.0` | Seconds |
| `fetch_timeout` | `20.0` | Seconds |
| `max_retries` | `2` | Retries after the first attempt |
| `backoff_factor` | `0.5` | Base of the exponential backoff |
| `max_bytes` | `2_000_000` | Body cap per page; oversized pages truncate |
| `max_concurrency` | `8` | Parallel fetches |
| `max_redirects` | `5` | Hops per fetch |
| `include_html` | `True` | Keep raw HTML on the `Page` |
| `include_images` | `True` | Collect images |
| `max_images` | `50` | Cap per page |
| `user_agent` | Chrome UA | Sent on every request |
| `headers` | `None` | Extra headers, merged in |
| `proxy` | `None` | e.g. `http://127.0.0.1:8118` |
| `verify_ssl` | `True` | TLS verification |
| `http2` | `True` | **Leave it on** — see below |
| `allow_private_networks` | `False` | See Security |
| `allowed_hosts` | `()` | Hosts exempt from the SSRF check |
| `cache_ttl` | `0` (off) | Seconds to reuse an identical search or fetch |
| `cache_size` | `256` | Cache entries |
| `auto_setup` | `True` | Start SearXNG on first `search()` if none is up |
| `setup_dir` | `~/.searchnow/searxng` | Where auto-setup puts the stack |

Per call, `search()` also takes `max_results`, `engines`, `categories`,
`language`, `safesearch`, `time_range`, `page`, `include_content`, `max_chars`,
`timeout`; and `fetch()` takes `timeout`, `include_html`, `include_images`,
`max_chars`, `raise_on_error`.

---

## What you get back

### `Page`

| | |
|---|---|
| `.ok` | Did it work. `Page` is truthy when it did |
| `.text` | Readable markdown — nav, cookie banners and footers gone |
| `.html` | Raw HTML source |
| `.images` | `list[Image]`, the page's own preview image first |
| `.main_image` | That preview image's URL, or `None` |
| `.title` `.description` `.author` `.published_date` `.sitename` `.language` | Metadata |
| `.url` `.final_url` `.status_code` `.content_type` | Where it came from |
| `.error` `.elapsed` `.truncated` `.fetched_at` | Diagnostics |
| `.to_dict()` | JSON-ready |

### `Image`

`.url` (always absolute), `.alt`, `.width`, `.height`, `.is_main`.

Images are found in `og:image`, `twitter:image`, JSON-LD, `<picture><source
srcset>` and every `<img>` — including lazy-loading attributes like `data-src`,
because on a modern page `src` alone is usually a placeholder spinner. `srcset`
resolves to the **largest** candidate. Tracking pixels and `data:` URIs are
dropped, and everything is deduplicated.

### `SearchResponse`

Iterable and indexable. `.results`, `.urls`, `.pages`, `.answers`,
`.suggestions`, `.corrections`, `.infoboxes`, `.unresponsive_engines`,
`.number_of_results`, `.elapsed`, `.cached`, `.to_dict()`, `.to_context()`.

### `SearchResult`

`.url`, `.title`, `.content` (engine snippet), `.engine`, `.engines`, `.score`,
`.category`, `.published_date`, `.thumbnail`, `.page`, `.raw`.

---

## Accuracy notes

The things that make the difference between "works on example.com" and "works
on real sites":

- **HTTP/2 is on by default.** Wikipedia, Cloudflare-fronted sites and others
  answer **403 to an HTTP/1.1 request** that claims to be a modern browser —
  because a real Chrome would never speak 1.1 to them. This one setting is the
  difference between a 403 and a 158 KB article on Wikipedia.
- **Two-pass text extraction.** trafilatura runs in precision mode first; if
  that returns suspiciously little, it retries in recall mode before falling
  back to flattening the document. Short pages stop coming back empty.
- **Lazy-loaded images are found**, and `srcset` picks the largest variant.
- **Charset detection** tries the `Content-Type` header, then the meta tag, then
  UTF-8, and never raises on undecodable bytes.
- **Retries** use exponential backoff with jitter and honour `Retry-After`, on
  429/5xx/timeouts only. A 4xx is never retried.
- **Parsing runs off the event loop.** lxml over a multi-megabyte document is
  CPU-bound; left in the loop it would stall every other in-flight fetch.
- **Anti-bot walls are named, not swallowed.** A 403 from Cloudflare and a 403
  from a permission error look identical until you read the body, so SearchNow
  reads it and tells you which one you hit.

---

## Security

`fetch()` refuses loopback, private, link-local, multicast and reserved
addresses, and **re-checks every redirect hop**. URLs handed to this method are
often produced by a model, so they are attacker-influenced input — this is what
stops a prompt-injected `http://169.254.169.254/latest/meta-data/`, or a public
URL that 302s to it, from reading your cloud credentials.

```python
page = fr.fetch_sync("http://169.254.169.254/latest/meta-data/")
page.ok       # False
page.error    # 'URLNotAllowedError: ... resolves to the non-public address ...'
```

Your SearXNG host is exempt automatically. To reach other internal hosts, opt in:

```python
FetchResult(allowed_hosts=("wiki.internal",))     # one host
FetchResult(allow_private_networks=True)          # everything, use with care
```

---

## Errors

```
SearchNowError
├── ConfigurationError          bad constructor arguments
├── TransportError              DNS / TLS / refused / reset
│   └── TimeoutError
├── HTTPStatusError             non-2xx (carries .status_code)
├── SearchBackendError          the instance answered with something unusable
│   └── JSONFormatDisabledError 403 on format=json — the message says how to fix it
├── URLNotAllowedError          blocked by the SSRF guard
└── ContentTypeError            the body is not text
```

`fetch()` and `fetch_many()` convert these into `Page.error` instead of raising.
`search()` raises them.

---

## Requirements

Python 3.10+, Docker (for SearXNG). Depends on `httpx[http2]` and
`trafilatura`.

---

## Development

```bash
git clone https://github.com/sahiljain/searchnow
cd searchnow
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

make test          # unit tests, fully mocked, no network
make lint          # ruff + mypy --strict
make integration   # needs a live SearXNG
```

See [CONTRIBUTING.md](CONTRIBUTING.md).

---

## License

MIT — see [LICENSE](LICENSE).
