Metadata-Version: 2.4
Name: bot-guard-py
Version: 0.7.0
Summary: Dependency-free HTTP bot detection and application-layer flood mitigation for Python 3.9+
Author: bot-guard-py contributors
License-Expression: MIT
Keywords: bot-detection,ddos-mitigation,rate-limiting,security,http,scraper,headless
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.80; extra == "fastapi"
Provides-Extra: flask
Requires-Dist: flask>=2.0; extra == "flask"
Provides-Extra: django
Requires-Dist: django>=3.2; extra == "django"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: mypy>=1.8; extra == "test"
Dynamic: license-file

# bot-guard-py

**bot-guard-py 0.7.0** is a zero-runtime-dependency Python 3.9+ HTTP bot detection and application-layer flood mitigation package. It includes bounded per-client/global token buckets and an in-flight request cap, as well as configurable bot scoring and policy actions.

## Quick start

```python
from bot_guard import BotGuard

guard = BotGuard(threshold=0.6)
result = guard.inspect(request.headers, client_ip="203.0.113.10")
if result.action == "BLOCK":
    return Response("Forbidden", status=403)
```

For direct enforcement:

```python
from bot_guard import BotDetected, BotGuard
try:
    result = BotGuard().enforce(request.headers, client_ip="203.0.113.10")
except BotDetected as blocked:
    return Response("Forbidden", status=403)
```

## Policy controls

| Capability | API |
|---|---|
| Shadow mode | `BotGuard(mode="shadow")` — never blocks, reports `MONITOR` |
| Allowlist | `allow_user_agents=[...]`, `allow_ips=[...]` |
| Denylist | `deny_user_agents=[...]`, `deny_ips=[...]` |
| Custom weights | `weights={"cli_tool": 0.9, "consistency": 0.25}` |
| Progressive thresholds | `challenge_at`, `rate_limit_at`, `block_at` |
| Bounded input | `max_headers`, `max_value_length` |
| Behavioral/reputation hook | `reputation_provider=...` |
| Structured events | `event_sink=callable` |
| Counters | `metrics=Metrics()` |
| Privacy-aware correlation | `fingerprint(headers, salt=...)` |
| Local rate limiting | `InMemoryRateLimiter(limit=60, window=60)` |
| JavaScript proof-of-work | `ChallengeManager(secret).issue()` / `.render()` |
| Browser attestation | `issue_attestation()` / `verify_attestation()` |
| JS WebDriver probe | `BrowserSignalManager` + `BrowserSignalASGI` / `BrowserSignalWSGI` |
| Behavioral history | `BehaviorTracker()` via `behavior_provider=` |
| HTTP flood mitigation | `DDoSProtector` + ASGI/WSGI middleware |

Default actions are `ALLOW` below 0.4, `CHALLENGE` from 0.4, `RATE_LIMIT` from 0.5, and `BLOCK` from the configured bot threshold (0.6 by default). `BotResult.explain()` gives a readable decision report, while `matched_rules` gives stable rule identifiers.

## Detection layers

The core inspects CLI clients, headless browsers, Puppeteer, Playwright, Selenium, PhantomJS, scrapers, missing browser headers, proxy/tunnel headers, browser-header consistency, IP/CIDR allowlists and denylists, and optional reputation/behavior providers. The opt-in signed browser probe adds client-reported WebDriver and automation-global signals. The core performs no network calls and never stores raw headers.

The included `InMemoryReputation` is a short-lived, thread-safe process-local score store. For Redis, ASN, VPN, Tor, datacenter, or behavioral systems, implement the tiny provider interface:

```python
class MyProvider:
    def score(self, headers, client_ip=None):
        return 0.0  # normalized 0..1
```

For short-lived correlation, `fingerprint()` hashes selected normalized headers and never stores raw values. For production fleets, replace `InMemoryRateLimiter` with a shared Redis or gateway limiter; the local implementation is useful for single-process deployments and tests.

`ChallengeManager` creates an expiring HMAC-signed browser challenge. Its rendered page solves a small SHA-256 proof-of-work using Web Crypto; a valid proof is accepted once per manager process, then the application can issue a short-lived signed browser attestation. Multi-process deployments need a shared replay store if they require global one-time use. This proves only that a browser-like client executed JavaScript; it is not hardware-backed attestation and must not be treated as identity proof. `BehaviorTracker` adds short-lived burst, path-scan, and 404/5xx error-history signals; unknown status `0` is not treated as an error.

## Middleware

FastAPI/Starlette blocks actions by default. Supply `on_action` to implement a real challenge or application-specific rate limit; otherwise `CHALLENGE` returns 403 and `RATE_LIMIT` returns 429:

```python
from bot_guard.middleware.fastapi import BotGuardMiddleware
app.add_middleware(BotGuardMiddleware, guard=BotGuard())
```

### JavaScript automation probe

For browser pages you control, add the same-origin script to the shared HTML layout. It solves a short SHA-256 nonce challenge, reports `navigator.webdriver`, known Selenium/Playwright globals, and a small interaction count, then receives a signed `HttpOnly; SameSite=Lax` cookie. A positive WebDriver signal is a strong risk signal and is blocked on the next request by default. Missing JavaScript or missing interactions is neutral; the interaction count is not proof that a user is human.

```python
import os
from bot_guard import (BotGuard, BrowserSignalManager, BehaviorTracker,
                       ChallengeManager, DDoSProtector)
from bot_guard.middleware import (BotGuardASGI, BrowserSignalASGI,
                                  DDoSProtectionASGI)

# `verified_crawlers` is built from the refreshed official feeds above.
challenge = ChallengeManager(os.environ["BOT_GUARD_SECRET"],
                             challenge_ttl=60, attestation_ttl=300)
signals = BrowserSignalManager(challenge)
guard = BotGuard(trusted_crawler_verifier=verified_crawlers,
                 browser_signal_provider=signals,
                 behavior_provider=BehaviorTracker())
protector = DDoSProtector(trusted_crawler_verifier=verified_crawlers)

# Wrap your FastAPI/Starlette `app` at its ASGI boundary.
app = BotGuardASGI(
    DDoSProtectionASGI(BrowserSignalASGI(app, signals), protector),
    guard=guard,
)
```

Add `<script src="/__bot_guard/probe.js" defer></script>` to the shared page template. For WSGI, use the same order with `BotGuardWSGI(DDoSProtectionWSGI(BrowserSignalWSGI(app, signals), protector), guard)`. The probe runs only where you add it; a first page request can complete before telemetry returns, so still protect sensitive actions with authentication, authorization, CSRF defenses, and rate limits. The server-issued token and cookie are signed and peer-IP-bound, but browser telemetry is still client-reported and can be hidden by stealth tools. Verified search-crawler CIDRs are checked before this signal, so verified crawlers do not need JavaScript. Behind a reverse proxy, configure all layers to use the same trusted client-IP resolver; never trust arbitrary forwarded headers.

### HTTP flood mitigation (application layer)

Use the separate rate/concurrency middleware **before the application does expensive work**:

```python
from bot_guard import DDoSProtector
from bot_guard.middleware.fastapi import DDoSProtectionMiddleware

protector = DDoSProtector(
    per_client_rate=20, per_client_burst=60,
    global_rate=1000, global_burst=2000,
    max_concurrent=500,
)
app.add_middleware(DDoSProtectionMiddleware, protector=protector)
```

For Flask, wrap the WSGI boundary: `from bot_guard.middleware.flask import init_ddos_protection; init_ddos_protection(app, protector)`. The generic equivalents are `DDoSProtectionASGI` and `DDoSProtectionWSGI`; Django provides `bot_guard.middleware.django.DDoSProtectionMiddleware`. `DDoSProtector` defaults to 20 requests/second per client (burst 60), 1,000 requests/second globally (burst 2,000), and at most 500 concurrent application requests. Tune these values to the site's real capacity; NAT/shared-IP users can share one per-client bucket. Rate rejections return 429; concurrency shedding returns 503.

### Search crawler handling

Do **not** allow `Googlebot`, `bingbot`, `DuckDuckBot`, or `DuckAssistBot` from the User-Agent string alone; any client can spoof it. Use `VerifiedCrawlerAllowlist.from_json_feeds()` with cached, current official feeds, then pass the same verifier to bot detection and flood control:

```python
import json
from pathlib import Path
from bot_guard import BotGuard, DDoSProtector, VerifiedCrawlerAllowlist
from bot_guard.middleware.fastapi import BotGuardMiddleware, DDoSProtectionMiddleware

# Download and validate these files outside the request path; refresh periodically.
# python examples/update_crawler_feeds.py --directory ./crawler-feeds
def read_feed(filename):
    return json.loads(Path(filename).read_text(encoding="utf-8"))

# Download/cache these feeds outside the request path; refresh as vendors update them.
verified_crawlers = VerifiedCrawlerAllowlist.from_json_feeds({
    "googlebot": read_feed("common-crawlers.json"),
    "bingbot": read_feed("bingbot.json"),
    "duckduckbot": read_feed("duckduckbot.json"),
    "duckassistbot": read_feed("duckassistbot.json"),
})
guard = BotGuard(trusted_crawler_verifier=verified_crawlers)
protector = DDoSProtector(trusted_crawler_verifier=verified_crawlers)

# `app` is your FastAPI application. Add flood control last so it is outermost.
app.add_middleware(BotGuardMiddleware, guard=guard)
app.add_middleware(DDoSProtectionMiddleware, protector=protector)
```

Verified crawlers skip heuristic bot blocking and the per-client rate bucket, but remain under the global rate/concurrency caps. Maintain feeds from [Google's official crawler ranges](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), [Bing's verification guidance/IP list](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26), [DuckDuckBot's official IP JSON](https://duckduckgo.com/duckduckbot.json), and [DuckAssistBot's official IP JSON](https://duckduckgo.com/duckassistbot.json). DuckDuckGo publishes each bot's User-Agent and IP list in its [DuckDuckBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) and [DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) documentation. Refresh feeds outside the request path and do not ship a stale baked-in list. DuckDuckBot respects `robots.txt`; DuckDuckGo documents opting out of DuckAssistBot via its robots.txt User-Agent. Do not use broad `allow_user_agents=["googlebot"]` (or other crawler names) for public traffic.

Flask and Django adapters are available in `bot_guard.middleware.flask` and `bot_guard.middleware.django`. For framework portability, use `BotGuardASGI` or `BotGuardWSGI` from `bot_guard.middleware.generic`. All adapters pass the socket peer IP to the detector and preserve the result on the request scope/object where the framework supports it. They do not trust forwarded headers; configure trusted-proxy handling in the application/server separately. Flask retains the `on_bot` callback name for compatibility and also accepts `on_action`.

## CLI

Inspect a JSON header object from a file or stdin:

```bash
echo '{"User-Agent":"curl/8.0"}' | bot-guard inspect -
echo '{"User-Agent":"curl/8.0"}' | bot-guard explain -
```

The CLI exits with status 1 for a blocking decision and status 0 otherwise, making it useful in gateways and smoke tests.

## Operations and security

Use `mode="shadow"` before enabling bot blocking on unfamiliar traffic. Send `event_sink` events to your logger or telemetry system without recording raw header values; `Metrics.snapshot()` exposes counters for export to Prometheus or OpenTelemetry adapters. Configure the reverse proxy to strip user-supplied forwarding headers and pass only trusted client IP values. The built-in token buckets are **process-local**: they do not coordinate across workers, containers, or regions. Supply shared `client_limiter` and `global_limiter` objects implementing `consume(key) -> (allowed, retry_after_seconds)` for distributed rate limiting, or enforce limits at a gateway/CDN.

This package can shed excess HTTP requests after they reach the application server. It **cannot stop network/volumetric DDoS or protect an already saturated link/origin**; keep CDN, WAF, hosting-provider DDoS protection, origin firewalling, and upstream rate limits in front. Header signals remain spoofable. Use authentication, quotas, and application authorization for protected resources. See [SECURITY.md](SECURITY.md) and [CHANGELOG.md](CHANGELOG.md).

## Development

```bash
python -m unittest discover -s tests -v
python -m compileall -q bot_guard
python -m pip wheel . --no-deps
# Optional real GUI Chromium/Selenium flow (requires Xvfb, Selenium, Chromium)
BOT_GUARD_RUN_GUI_TESTS=1 xvfb-run -a python -m pytest -q
```

CI covers Python 3.9 through 3.13, compilation, tests, and wheel building. MIT licensed.
