Metadata-Version: 2.5
Name: oss-research-web
Version: 0.3.0a1
Summary: Independent SearXNG search and bounded page extraction for research agents
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4<5,>=4.13
Requires-Dist: httpx<0.29,>=0.28
Requires-Dist: markdownify<2,>=1.2
Requires-Dist: pillow<13,>=12.1
Requires-Dist: pypdf<7,>=6
Provides-Extra: browser
Requires-Dist: playwright<2,>=1.55; extra == 'browser'
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: pytest-asyncio<2,>=1; extra == 'dev'
Requires-Dist: pytest<10,>=8; extra == 'dev'
Requires-Dist: ruff<1,>=0.12; extra == 'dev'
Description-Content-Type: text/markdown

# OSS Research Web (experimental)

Standalone Python package. Search (`SearXNGSearch`) and extraction (`PageReader`)
are independently usable; `ResearchSession` adds per-agent references/open/find.
No Modis, Codex, model provider, API key, or paid fallback is required.
SearXNG is an external service, not bundled Python code.

This is an experimental prerelease. Install with
`pip install oss-research-web==0.3.0a1`; deployment still requires a SearXNG service.
See [v0.3 image validation](validation/REPORT-v0.3.md), [v0.2 validation](validation/REPORT-v0.2.md),
[initial validation](validation/REPORT.md), and validation/ACCEPTANCE.md for evidence.
The current candidate uses the keyless Google and Yahoo SearXNG adapters, not
Google's paid API. Engine availability is environment-dependent.

## Development

```
uv sync --extra dev --extra browser
uv run pytest
uv run ruff check .
uv run python -m build
uv run playwright install chromium
```

Linux is currently required for the parser process resource limits. The tested
interpreter was Python 3.13.9. A supplied browser executable can also be selected.
PDF screenshots additionally require Poppler's `pdftoppm` executable on PATH
(for example, the `poppler-utils` OS package). Search and text extraction work
without Poppler; rendering reports `renderer_unavailable` if it is absent.

Version 0.3.0a1 adds page figures, image search, bounded decoding, original-pixel
crops, explicit previews, PDF crops and multimodal input content to the existing
filtered search and source-snapshot workflows. See [image usage](docs/IMAGES.md).
It does not enable this provider in deployed agents.

## Reproduce the live evaluation

From this package directory (not the oss-agent deployment directory):

```
docker compose -f deploy/compose.yaml up -d
uv run python validation/run.py --output validation/runs/CHOOSE-A-NEW-RUN-NAME
uv run python validation/features.py --output validation/runs/CHOOSE-ANOTHER-NEW-RUN-NAME
docker compose -f deploy/compose.yaml down
```

The runner refuses an existing output directory. It makes 30 sequential searches,
20 different searches with four concurrent callers, and ten independent page
reads. Requests are paced to one start/second across a shared provider. This
launches no model and consumes no Valyu credits. Do not reuse the old run names.

For the browser fixture tests, set `RUN_BROWSER_TESTS=1`; also set
`PLAYWRIGHT_BROWSERS_PATH` if Chromium was installed in a non-default location.

The dependency lock is for reproducible validation. Browser installation is
optional for HTTP/HTML/PDF use. Run browser work in a container when appropriate.

## API

```python
import asyncio
from research_web import SearXNGSearch, PageReader, ResearchSession, SearchFilters


async def main():
    async with SearXNGSearch("http://127.0.0.1:18991") as search, PageReader() as reader:
        session = ResearchSession(search, reader)
        results = await session.search(
            "continual learning reservoir replay", limit=5,
            filters=SearchFilters(allowed_domains=("arxiv.org", "proceedings.neurips.cc")),
        )
        page = await session.open(results.results[0].url)
        print(page.view(start=1, lines=80))
        print(session.find("replay"))
        print(session.sources())


asyncio.run(main())
```

`SearchResponse` distinguishes successful empty results, degraded engines, and
exceptions. `WebError.code` is machine readable. Sources carry stable URL-derived
IDs. Pages carry a content hash, final URL, fetch timestamp and extraction method.
Line slices are one-based. Find is literal text, with bounded matches/context;
untrusted regex is deliberately not executed.

An async event sink can receive started/completed/failed/cache_hit events with
operation IDs. A future Modis adapter can translate these without provider-specific
logic. Cache entries are immutable and bounded by count, bytes and TTL. Session
references/current page are never shared between agents.

HTTP extraction keeps main/article content when available, strips navigation and
scripts, preserves code/tables/links, and resolves relative links. Browser fallback
is optional and only used for sparse HTML; it is not CAPTCHA or paywall bypass.
PDF extraction is text-only, page-labelled, with no OCR or layout accuracy promise.
Optional PDF screenshots preserve visual layout for inspection; they do not
transcribe it or make the model using this library capable of vision.

## Research navigation and provenance

See [the workflow contract](docs/RESEARCH_WORKFLOW.md) for the full API and limits.
`Page.source_id` identifies a URL; `Page.snapshot_id` identifies a particular
extracted representation and original-byte hash. `open(snapshot_id)` reuses that
exact retained version. `open(url, refresh=True)` fetches a new version. An expired
snapshot fails explicitly rather than silently refetching a possibly changed page.
`sources()` distinguishes search-only discoveries from opened, hashed documents.
`links(snapshot_id)` lists numbered links; `click(id, target=snapshot_id)` follows
one. `find(text, target=snapshot_id)` searches an earlier document without changing
the current page. `view()` returns bounded excerpts and continuation positions,
including an offset when one line exceeds the character budget. Full retained
text remains available; a view limit never discards the remainder of a document.

```python
paper = await session.open("https://arxiv.org/pdf/1706.03762")
print(paper.pdf_text(start_page=3, end_page=4))
image = await session.screenshot(paper.snapshot_id, page=3, max_edge=1600)
# image.png is a PNG byte string; image.data_url() is suitable for a vision input.
# PDF page numbers and link IDs are one-based throughout this package.
```

For image-only PDFs, text extraction still reports `no_text`. Use `pdf_info(url)`
and `screenshot(url, page=...)` to inspect them visually. OCR and automatic
model/gateway tool integration remain separate work. General image search and HTML
image extraction are now available through the explicit [image APIs](docs/IMAGES.md).

Public page requests reject credentials, local/private destinations and non-HTTP
schemes; DNS is resolved and pinned per request, with redirects revalidated. Only
GET is supported. Browser subresources use the same fetcher; service workers and
WebSockets are blocked. SearXNG's configured local endpoint is explicitly trusted.
The prototype is a library, not a multi-tenant security boundary. Use a restricted
container for browser execution and untrusted document parsing in production.

Validation uses a separate SearXNG container; no agent profile or active gateway is
modified. Publishing and integration remain gated by the independent evidence.
