Metadata-Version: 2.4
Name: forgemesh-pay-per-crawl
Version: 0.1.2
Summary: Charge AI crawlers and bots per page with x402 USDC micropayments on Base. ASGI, WSGI and Django middleware; you keep 100%.
Author-email: ForgeMesh Labs <hello@forgemesh.io>
License-Expression: MIT
Project-URL: Homepage, https://forgemesh.io
Project-URL: Repository, https://github.com/forgemeshlabs/pay-per-crawl-py
Project-URL: Issues, https://github.com/forgemeshlabs/pay-per-crawl-py/issues
Project-URL: PHP-WordPress-edition, https://github.com/forgemeshlabs/forgemesh-pay-per-crawl
Keywords: x402,mpp,pay-per-crawl,402,ai-crawlers,usdc,base,fastapi,flask,django,asgi,wsgi
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pycryptodome>=3.20
Provides-Extra: httpx
Requires-Dist: httpx>=0.27; extra == "httpx"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: httpx>=0.27; extra == "test"
Requires-Dist: fastapi>=0.110; extra == "test"
Requires-Dist: uvicorn>=0.29; extra == "test"
Requires-Dist: flask>=3; extra == "test"
Requires-Dist: django>=4.2; extra == "test"
Dynamic: license-file

# ForgeMesh Pay-per-Crawl for Python

Charge AI crawlers and bots per page with [x402](https://x402.org) USDC micropayments on Base, paid straight to your wallet. Humans and verified search engines read free. Middleware for **FastAPI / Starlette (ASGI)**, **Flask and any WSGI app**, and **Django**, on one framework-free core.

This is the Python edition of the [WordPress plugin](https://github.com/forgemeshlabs/forgemesh-pay-per-crawl); behaviour is identical.

- **Humans see nothing different.** Browsers get the page as always.
- **Verified search engines stay free.** Googlebot, Bingbot, DuckDuckBot, Applebot and others are checked by reverse DNS, so SEO is untouched. A crawler that only *claims* to be Googlebot is treated as a script.
- **AI crawlers, SEO crawlers and scripts get `402 Payment Required`** with an x402 challenge (v2 `PAYMENT-REQUIRED` header + v1 JSON body) and an MPP `WWW-Authenticate: Payment` challenge. They pay USDC on Base and retry; the page is served only after the payment settles on-chain.
- **You keep 100%.** The payer's USDC goes directly to your wallet. No custody, no account, no API key, no revenue share.
- **Discovery built in.** Serves `/.well-known/x402.json`, `/llms.txt` and `/agents.txt`.

## Install

```
pip install forgemesh-pay-per-crawl
```

Python 3.10+. One dependency (`pycryptodome`, for keccak256). `httpx` is used for facilitator calls if installed, otherwise the standard library. Start with `mode="monitor"` (records what bots would have paid, never blocks), then switch to `mode="gate"`.

## FastAPI / Starlette

```python
from forgemesh_pay_per_crawl import Config
from forgemesh_pay_per_crawl.asgi import PayPerCrawlMiddleware

app.add_middleware(PayPerCrawlMiddleware,
    config=Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
```

## Flask (and any WSGI app)

```python
from forgemesh_pay_per_crawl import Config
from forgemesh_pay_per_crawl.wsgi import PayPerCrawlWSGI

app.wsgi_app = PayPerCrawlWSGI(app.wsgi_app,
    Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
```

## Django

```python
# settings.py
MIDDLEWARE = ["forgemesh_pay_per_crawl.django.PayPerCrawlMiddleware", *MIDDLEWARE]  # near the top
PAY_PER_CRAWL = {"pay_to": "0xYourWalletOnBase", "mode": "gate", "price_usd": 0.01, "base_url": "https://example.com"}
```

## Settings

All are `Config(...)` keyword arguments (or keys of `PAY_PER_CRAWL`).

| Setting | Default | Meaning |
|---|---|---|
| `pay_to` | required in gate mode | Your `0x` wallet on Base. |
| `mode` | `"monitor"` | `"monitor"` records only; `"gate"` answers 402. |
| `price_usd` | `0.01` | Price per page fetch, paid in USDC. |
| `facilitator_url` | PayAI (`https://facilitator.payai.network`, no key) | Any x402 facilitator that supports Base. `facilitator_auth` sets an `Authorization` header if yours needs one. |
| `class_policy` | search/social/monitor free; seo/ai/automation/bot paid | Per class: `"free"`, `"paid"` or `"block"`. |
| `bot_overrides` | `{}` | Per bot name, e.g. `{"GPTBot": "free", "Bytespider": "block"}`. |
| `allow_paths` | `[]` | Path prefixes that are never gated. `/robots.txt`, `/sitemap*`, `/llms.txt`, `/agents.txt`, `/.well-known/`, `/favicon.ico` always are. |
| `base_url` | from the request | Your public origin. Set it behind a proxy or load balancer. |
| `trust_proxy_header` | `""` | Header with the real client IP (`"CF-Connecting-IP"`, `"X-Forwarded-For"`) so search-engine verification sees the right address. |
| `mpp_secret` | random per process | HMAC key for MPP challenge ids. Set a stable value if you run more than one worker. |
| `gated_methods` | `("GET", "HEAD")` | HEAD is gated like GET (402 with no body). |
| `site_name`, `site_description`, `pages` | | Used in the discovery files; `pages` is a list (or callable returning one) of `(title, url)` for `llms.txt`. |
| `on_event` | `None` | Callback receiving a dict for every classified request (status, bot, class, rail, tx hash, payer). |

Bot classes: `human`, `search`, `social`, `monitor`, `seo`, `ai`, `automation`, `bot`. The catalog covers 85 crawlers and tools; see `core/catalog.py`.

Multiple worker processes: pass `lock=FileNonceLock("/var/tmp/fmppc")` to `Gate(...)` (and use `PayPerCrawlMiddleware(app, gate=gate)`) so a payment authorization cannot be replayed on another worker, and set a stable `mpp_secret`. The default lock is in-process memory.

### Behind Cloudflare

If your site is proxied by Cloudflare, set **AI Crawl Control** (and any Bot Fight Mode / AI Labyrinth / "Block AI bots" rule) to **Allow** for the crawlers you want to charge. Cloudflare answers those requests at the edge before they ever reach your app, so the gate would never see them. Set `trust_proxy_header="CF-Connecting-IP"` and `base_url` to your public origin.

## How a payment works

1. A bot requests a page and gets `402` with the price, asset (USDC on Base) and your wallet, in the `PAYMENT-REQUIRED` header (x402 v2), the JSON body (x402 v1, with `invalidReason` when a payment was rejected) and `WWW-Authenticate: Payment` (MPP).
2. The bot signs a USDC transfer authorization (EIP-3009) and retries with it in `PAYMENT-SIGNATURE` (v2), `X-PAYMENT` (v1) or `Authorization: Payment` (MPP).
3. The middleware checks recipient, amount and expiry locally, locks the authorization nonce against replay, then posts it to the **facilitator** (`/verify`, then `/settle`). The facilitator verifies and submits it on-chain; it never holds funds. v1 payments are forwarded in v1 shape.
4. The page is served only when a transaction hash comes back, with the receipt in `PAYMENT-RESPONSE`, `X-PAYMENT-RESPONSE` and (MPP) `Payment-Receipt`.

The package never contacts ForgeMesh. Its only outbound requests are to the facilitator you chose, and only when a payment is presented. (The optional Pro reporter below is the one exception, and is off without a key.)

## Money flow

The payer's USDC moves wallet to wallet on Base, from the payer to the `pay_to` address you configured. There is no ForgeMesh account, no custody, no settlement service and no revenue share: the site owner keeps 100%. The facilitator (PayAI by default) charges the seller no fee at the time of writing; check your facilitator's terms.

## Using the pieces directly

```python
from forgemesh_pay_per_crawl import Classifier, Config, Gate, Request, X402, build_challenge

Classifier().classify("Mozilla/5.0 (compatible; GPTBot/1.2)").cls        # "ai"
gate = Gate(Config(pay_to="0x...", mode="gate"))
decision = gate.handle(Request(method="GET", path="/x", headers={"user-agent": "curl/8"}, host="example.com"))
decision.response   # a Response (status, headers, body) to send, or None to serve the page
decision.headers    # headers to add to the page you serve (payment receipt, Link)
```

`Gate.handle` is blocking (it can call the facilitator); in async code run it in a thread, as the ASGI middleware does. The facilitator transport is injectable (`Gate(config, transport=fn)` with `fn(url, body, headers) -> dict`).

## Pro (optional)

With a ForgeMesh Pay-per-Crawl Pro license (`pro_key="fmppc_..."`) the package reports daily bot rollups and settlements to the hosted dashboard at proxy.forgemesh.io and pulls the crawler-intel feed into the classifier. Nothing is sent without a key. The rollups are held in memory per process, so with several workers each reports its own counts.

## Verified

- Unit and integration tests: `pytest` (keccak vectors incl. block-boundary cases, classifier with rDNS, 402 shapes, MPP challenge/credential round trip, facilitator flow, replay, HEAD, file lock, Pro sync, all three adapters, and an integration matrix against live uvicorn/FastAPI and Flask servers with a mock facilitator).
- Real settlements on Base mainnet through PayAI, October 2026, from the FastAPI example gated at $0.01: x402 v1 ([tx](https://basescan.org/tx/0x49f0f05b3509545e5134aa6cf091ff7eab4e4f1e1b9aef9bf28280c231e2a945)) and x402 v2 ([tx](https://basescan.org/tx/0x15d95ad90c9530e4816016f65681ddd4a79730373634c991c57afa2b88a8b44b)).

## Development

```
python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest
python -m build
```

MIT. Made by [ForgeMesh Labs](https://forgemesh.io) (hello@forgemesh.io).
