Metadata-Version: 2.4
Name: webeater
Version: 0.2.2
Summary: A web content extraction tool designed to fetch and process web pages efficiently
Author-email: Tiago Ribeiro <webeater@tiagoribeiro.pt>
Maintainer-email: Tiago Ribeiro <webeater@tiagoribeiro.pt>
License-Expression: MIT
Project-URL: Homepage, https://github.com/tiagrib/webeater
Project-URL: Repository, https://github.com/tiagrib/webeater.git
Project-URL: Issues, https://github.com/tiagrib/webeater/issues
Project-URL: Documentation, https://github.com/tiagrib/webeater#readme
Keywords: web,scraping,extraction,selenium,beautifulsoup,content
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: annotated-types>=0.7.0
Requires-Dist: attrs>=25.3.0
Requires-Dist: beautifulsoup4>=4.13.4
Requires-Dist: bs4>=0.0.2
Requires-Dist: certifi>=2025.8.3
Requires-Dist: cffi>=1.17.1
Requires-Dist: coloredlogs>=15.0.1
Requires-Dist: curl-cffi>=0.7
Requires-Dist: h11>=0.16.0
Requires-Dist: httpx>=0.27
Requires-Dist: httpx-curl-cffi>=0.1
Requires-Dist: humanfriendly>=10.0
Requires-Dist: idna>=3.10
Requires-Dist: outcome>=1.3.0.post0
Requires-Dist: playwright>=1.45
Requires-Dist: pycparser>=2.22
Requires-Dist: pydantic>=2.11.7
Requires-Dist: pydantic_core>=2.33.2
Requires-Dist: pyreadline3>=3.5.4
Requires-Dist: PySocks>=1.7.1
Requires-Dist: selenium>=4.34.2
Requires-Dist: sniffio>=1.3.1
Requires-Dist: sortedcontainers>=2.4.0
Requires-Dist: soupsieve>=2.7
Requires-Dist: trio>=0.30.0
Requires-Dist: trio-websocket>=0.12.2
Requires-Dist: typing-inspection>=0.4.1
Requires-Dist: typing_extensions>=4.14.1
Requires-Dist: urllib3>=2.5.0
Requires-Dist: websocket-client>=1.8.0
Requires-Dist: wsproto>=1.2.0
Requires-Dist: zendriver>=0.15
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: psutil>=5.9; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=6.0; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Dynamic: license-file

<img src="img/logo.png" alt="Logo" style="max-height: 100px;">

# WebEater (weat)

WebEater is a Python library for backend services that need to fetch
structured content (Markdown text or a JSON dict) from arbitrary URLs as
fast as possible. It exposes a small async API (`Webeater.create`,
`engine.get`, `engine.get_many`, `engine.shutdown`) designed to be built
once at service startup and reused across many requests. A `weat` /
`webeater` CLI ships in the same package for one-shot extraction and
interactive use, but the library is the primary target.

## Main Features

- Library + CLI in one package — same engine, two surfaces.
- Hybrid HTTP-first renderer with lazy escalation to a JS engine
  (`zendriver` default, `playwright` opt-in, `selenium` legacy). Static
  pages skip the browser boot entirely. The default renderer chain
  (`hybrid` + `zendriver` + `bs`) was selected by the project's
  pipeline matrix bench (`python tests/bench_all.py --preset realistic`)
  on 2026-05-18 as the best-on-average pipeline across the realistic URL
  pool.
- One supported extractor: `WebeaterBeautifulSoup` (default). The
  experimental `WebeaterFastBS` is deprecated as of 2026-05-24 (D16)
  and scheduled for removal in v0.3.0 — see "Extractor selection" below.
- Output as Markdown text or a structured JSON dict (`title`, `content`,
  `images`, `links`, `fetch_time`).
- Concurrent batch fetch via `Webeater.get_many(urls, max_concurrent=N)`
  (single engine) or `WebeaterPool` (N engines, true cross-engine
  parallelism for SPA-heavy workloads).
- Layered hints system for noise removal and main-content selectors.
- Tested on Python 3.9 through 3.12.

## Lifecycle and usage patterns

How you use webeater drives which costs dominate. The **warm** pattern
(build the engine once, reuse it across many `get()` calls) saves the
multi-second cost of importing the JS engine (zendriver by default) and
booting the browser context per request — this is the recommended shape
for backend services. The **cold** pattern (one full lifecycle per
invocation) pays those costs every time and is the natural fit for the
CLI or for serverless invocations where state cannot be kept.

See `metak-shared/architecture.md` ("Lifecycle and usage patterns") for
the full breakdown, concurrency notes, and per-request data flow.

## Install

```
pip install webeater
```

The default JS engine is `zendriver`, which uses the host's installed
Chrome — no separate browser-binary install step. If you want
Playwright instead (`WeatConfig.js_engine="playwright"`), install its
browser binaries with:

```
playwright install chromium
```

Selenium also remains supported and uses the host's installed Chrome
plus Selenium Manager (no extra install step). See "JavaScript engine"
below for how to switch engines.

## Quick Start (library, recommended pattern)

Build the engine once at process startup. Reuse it across requests.
This is the cost model that matters for FastAPI apps, worker
processes, daemons, and any long-running service.

```python
import asyncio
from webeater import Webeater


async def main():
    # Build once. Multi-second boot cost paid here.
    engine = await Webeater.create()
    try:
        for url in [
            "https://en.wikipedia.org/wiki/Python_(programming_language)",
            "https://docs.python.org/3/library/asyncio.html",
            "https://news.ycombinator.com/",
        ]:
            result = await engine.get(url, return_dict=True)
            print(result["title"], "—", result["fetch_time"])
    finally:
        # Releases Chromium and the httpx client.
        await engine.shutdown()


asyncio.run(main())
```

For concurrent batches against a single engine, use
`engine.get_many(urls, max_concurrent=8)` — the HTTP fast path runs in
parallel, and the JS engine path is serialised internally so one
browser context stays safe. For true cross-engine concurrency (e.g.
SPA-heavy workloads where every request escalates to the JS engine),
use `WebeaterPool` — see the "Backend integration" section below.

## Quick Start (one-off / CLI use)

For a single URL with no service to keep state in, the cold pattern is
fine — but be aware that **every invocation pays the full import +
browser-boot cost**, which usually dwarfs the actual fetch.

```python
import asyncio
from webeater import Webeater


async def main():
    engine = await Webeater.create()
    try:
        print(await engine.get("https://example.com"))
    finally:
        await engine.shutdown()


asyncio.run(main())
```

For genuine one-off invocations from a shell, the `weat` / `webeater`
CLI (see next section) is the natural fit — same engine, no Python
boilerplate.

## Quick Start (CLI)

```
weat https://example.com
```

Fetches the page and prints the extracted text to the console.

### CLI Options

- url (positional): URL to fetch. If omitted, starts an interactive prompt.
- -c, --config FILE (default: weat.json): Config file.
- --hints FILE [FILE ...]: Additional hint files (space-separated).
- --debug: Enable debug logging.
- --silent: Suppress debug/info; print only result or errors (for scripts).
- --json: Return content as JSON instead of plain text.
- --content-only: Return only the main extracted content (skip images/links).

```
webeater https://example.com
webeater --json --content-only https://example.com
webeater -c weat.json --hints hints/news.json hints/sports.json https://example.com
```

Interactive mode (when no URL is provided): enter a URL when prompted.
Per-request prefix shortcuts: `j!<url>` (JSON), `c!<url>` (content
only), `jc!<url>` or `cj!<url>` (JSON + content only). `q` quits.
URLs must start with `http://` or `https://`.

## Help and Contributions

For questions or discussions, start a [Discussion](https://github.com/tiagrib/webeater/discussions).
For bugs or contributions, open an [Issue](https://github.com/tiagrib/webeater/issues).

## Develop with Source

Clone the repository at <https://github.com/tiagrib/webeater.git>, then
install the development dependencies:

```
pip install -r requirements.txt
```

Tested on Python 3.9 through 3.12 (daily on 3.12.3).

## Configuration and Advanced documentation

WebEater uses a `weat.json` configuration file (in the current working
directory) to manage its settings. The file is read on construction and
written back after every load, with defaults omitted to keep it clean.

For detailed documentation on the hints system, see the
[Hints Documentation](hints/README.md).

### Backend integration

**For daemons and concurrent-request services, see
[`INTEGRATION.md`](INTEGRATION.md) — it covers the warm-pool /
`WeatService` pattern end-to-end (FastAPI, queue consumers,
sync bridges, shutdown, telemetry, sizing, and what not to do).** The
section below is a brief introduction to the lower-level `WebeaterPool`
primitive that `WeatService` is built on.

For a long-running service (FastAPI / Starlette / aiohttp / a worker
daemon), build a `WebeaterPool` once at startup and tear it down at
shutdown. Each pool member holds its own browser context, so requests
that escalate to the JS engine run in parallel up to the pool size.

```python
import asyncio
from fastapi import FastAPI
from webeater import WebeaterPool

app = FastAPI()
pool: WebeaterPool | None = None


@app.on_event("startup")
async def startup():
    global pool
    pool = await WebeaterPool.create(size=4)


@app.on_event("shutdown")
async def shutdown():
    if pool is not None:
        await pool.shutdown()


@app.get("/fetch")
async def fetch(url: str):
    assert pool is not None
    return await pool.get(url, return_dict=True)
```

Pool size should be tuned to expected concurrency. Each pool member
holds one browser context, so memory scales linearly with pool size.
`size=4` is a reasonable default for a single FastAPI worker.

For a single shared engine without per-request parallelism through the
JS engine, the simpler `Webeater.create()` + `engine.get()` pattern
still works — but `WebeaterPool` is the recommended shape for backend
services because the JS-engine escalation path is serialised inside a
single engine and saturates quickly under load.

### Service: concurrent request dispatch (`WeatService`)

`WeatService` (added in v0.2.0) sits one level above `WebeaterPool` and
adds a request queue with per-request callbacks and futures, so you can
drive a warm pool of N engines from any transport — FastAPI, a Redis
queue consumer, a CLI batch processor, a thread bridge from sync code —
without writing your own dispatcher. Each `submit(url, ...)` returns a
`ServiceFuture` immediately; an optional callback fires when the
request completes.

```python
import asyncio
from fastapi import FastAPI, HTTPException
from webeater import WeatService

app = FastAPI()
svc: WeatService | None = None


@app.on_event("startup")
async def _startup() -> None:
    global svc
    svc = await WeatService.create(pool_size=4)


@app.on_event("shutdown")
async def _shutdown() -> None:
    if svc is not None:
        await svc.shutdown(drain=True)


@app.get("/fetch")
async def fetch(url: str) -> dict:
    assert svc is not None
    result = await svc.submit(url)
    if result.error is not None:
        raise HTTPException(502, str(result.error))
    return result.result
```

Fire-and-forget with a callback (queue-consumer pattern):

```python
from webeater import WeatService, ServiceResult


async def deliver(result: ServiceResult) -> None:
    await my_outbound_queue.put(result)


svc = await WeatService.create(pool_size=16, on_result=deliver, on_error=deliver)

async for url in incoming_urls:
    svc.submit(url)  # no await -- result delivered via callback
```

Bridging from sync code (thread):

```python
import asyncio
from webeater import WeatService

svc = await WeatService.create(pool_size=4)
loop = asyncio.get_running_loop()


def from_sync_thread(url: str):
    fut = asyncio.run_coroutine_threadsafe(svc.submit_async(url), loop)
    return fut.result(timeout=30)
```

Sizing: each pool member spawns a Chrome subprocess via the JS engine,
so memory scales roughly linearly with `pool_size`. See
`tests/bench_service_memory.py` (requires `pip install webeater[dev]`
for `psutil`) for the per-worker RSS sweep against your hardware. The
full surface — `submit` / `submit_async` / `submit_many` / `shutdown` /
telemetry properties / `ServiceResult` fields / callback semantics — is
documented in `metak-shared/api-contracts/service.md`.

### Extractor selection

Two content extractors, picked via `extractor` in `weat.json`:

- `bs` (default) — the BeautifulSoup-walking extractor. Stable, ships
  the production output shape, and is the documented default since
  2026-05-24 (D16).
- `fastbs` — **Deprecated (will be removed in v0.3.0).** Provided no
  measurable advantage over the `bs` extractor on real-world content
  and can occasionally clip the output or get gated by anti-bot
  responses. Still functional through v0.2.x; constructing
  `WeatConfig(extractor="fastbs")` emits a `DeprecationWarning`. Users
  should drop the `extractor` field from their `weat.json` (or set
  `extractor="bs"`) to silence the warning.

```
{"extractor": "fastbs"}
```

The default (`bs`) is omitted from the saved `weat.json`.

### Renderer selection

WebEater ships with four HTML renderers, picked via `renderer` in
`weat.json`:

- `hybrid` (default) — HTTP fast path via `httpx`, then an optional
  Chrome-impersonating HTTP middle tier (`curl_cffi`, on by default
  since T30 — see "Browser-impersonating HTTP middle tier" below),
  then lazy escalation to a JS-rendering engine when the response
  looks JS-rendered (empty body, SPA shell markers, SPA-framework
  shell with thin visible text, anti-bot challenge interstitial,
  very short visible text, or HTTP errors). Static pages skip the
  browser boot entirely. The escalation engine is chosen by
  `js_engine` (below).
- `zendriver` — always zendriver (async-first, CDP-direct,
  undetected-by-default) against the host's installed Chrome. Used as
  the default JS engine inside `hybrid` since 2026-05-18 (see
  "JavaScript engine" below).
- `playwright` — always headless Chromium via Playwright.
  Async-native and CDP-direct. Requires `playwright install chromium`
  to fetch browser binaries.
- `selenium` — always headless Chrome via Selenium. Legacy, kept for
  compatibility; slower than zendriver / Playwright.

```
{"renderer": "selenium"}
```

The default (`hybrid`) is omitted from the saved `weat.json`.

### JavaScript engine

When `renderer` is `hybrid`, `js_engine` picks which engine
`HybridRenderer` escalates to: `zendriver` (default — async-first,
CDP-direct, undetected-by-default, uses the host's Chrome install),
`playwright` (opt-in — async-native, CDP-direct, headless Chromium;
requires `playwright install chromium`), or `selenium` (legacy).

```
{"js_engine": "playwright"}
```

The default (`zendriver`) is omitted from the saved `weat.json`. The
flip from `playwright` to `zendriver` on 2026-05-18 is documented in
`metak-orchestrator/DECISIONS.md` (D9 amendment) and was driven by the
realistic-preset run of `tests/bench_all.py`.

### Browser-impersonating HTTP middle tier

When `renderer` is `hybrid`, a `curl_cffi`-backed middle tier between
`httpx` and the JS engine impersonates Chrome's TLS / JA3 fingerprint
to bypass sites that 403 plain `httpx` on TLS-fingerprint detection
alone (notably Wikimedia). On by default since T30 (2026-05-18) — the
new SPA-framework branch in `HybridRenderer._looks_js_rendered`
detects partially-hydrated SPA responses and escalates them to the JS
engine, closing the regression that previously held back this default.
Opt out via:

```
{"use_impersonate_tier": false}
```

### User-Agent override

All renderers impersonate a recent stable Chrome on Windows by default.
Override per config (`{"user_agent": "MyCrawler/1.0"}`) or programmatically
via `WeatConfig(user_agent="MyCrawler/1.0")`. Sites with TLS fingerprinting
plus JS challenges (Cloudflare-grade) still require the full-browser
escalation path, which the hybrid renderer already does on demand.

### Benchmarks

Standalone bench scripts under `tests/` — none are part of CI; run
manually. The project's success metric is **full end-to-end wall-clock
for one `engine.get(url)` request**; the full-query benches measure
exactly that. The diagnostic benches isolate individual components and
are useful for finding hot spots but should not be used to justify
default-affecting decisions (see `metak-shared/overview.md` "What 'fast'
means here").

Full-query benches (canonical — represent user-facing latency):

* `bench_pipeline_breakdown.py` — **canonical full-query bench.** Real
  per-iteration `render + extract` per URL. This is the bench whose
  numbers represent user-facing latency; run it with `--preset realistic`
  for the headline picture.
* `bench_end_to_end.py` — full-query across a config matrix (v0.1.1
  baseline vs current default, etc.).
* `bench_all.py` — unified entry point that runs the others and prints
  one summary. Wraps every bench listed in this section.
* `bench_renderers_live.py` — real side-by-side wall-clock per URL
  across every renderer option (no simulation). Opt in to running it
  from `bench_all.py` via `--include-renderer-live`.
* `bench_warm_vs_cold.py` — warm-vs-cold lifecycle cost. Measures
  `Webeater.create + get + shutdown` (cold) versus `get` on a pre-built
  engine (warm) per renderer config, so you can see how much per-request
  cost a backend service saves by keeping the engine alive. Opt in to
  running it from `bench_all.py` via `--include-warm-cold`.
* `bench_pool.py` — pool concurrency. Compares three batched paths
  against the same URL set: serial (one engine, one URL at a time),
  `Webeater.get_many` (one engine, concurrent), and
  `WebeaterPool.get_many` (N engines, true cross-engine concurrency).
  Opt in to running it from `bench_all.py` via `--include-pool`.

#### Diagnostic-only benches

These measure individual components in isolation. Useful for finding hot
spots, not for justifying default-affecting decisions — use the
full-query benches above for that (see `metak-shared/overview.md`).

* `bench_extractors.py` — extractor on a synthetic HTML fixture.
* `bench_extractors_live.py` — extractor on real HTML, fetched once and
  replayed in isolation.
* `bench_renderers.py` — architectural simulation: cost shapes of
  Selenium / Playwright escalations without launching browsers.
