Metadata-Version: 2.5
Name: agentpulse-asgi
Version: 0.1.0
Summary: AgentPulse for Python: detect AI agents, negotiate Markdown and declare Content Signals (FastAPI/Starlette).
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,asgi,content-signals,fastapi,llm,markdown,robots-txt,starlette
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: AsyncIO
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: asgi
Requires-Dist: starlette>=0.40; extra == 'asgi'
Description-Content-Type: text/markdown

# agentpulse-asgi

AgentPulse for Python: detect AI agents, serve them clean Markdown and declare
[Content Signals](https://contentsignals.org) in FastAPI, Starlette or any ASGI app.
Standard library only.

```bash
pip install agentpulse-asgi
```

## Usage

```python
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from agentpulse_asgi import AgentPulseMiddleware, ContentSignals, content_signals_robots_txt

signals = ContentSignals(search=True, ai_input=True, ai_train=False)

app = FastAPI()
app.add_middleware(AgentPulseMiddleware, content_signals=signals, on_event=print)


@app.get("/robots.txt", response_class=PlainTextResponse)
def robots() -> str:
    return content_signals_robots_txt(signals)
```

Requests that prefer `text/markdown` (RFC 9110 q-values) get the HTML page converted to Markdown;
everyone else gets the original HTML. HTML responses carry `Vary: Accept`. Markdown responses carry
`x-markdown-tokens` / `x-original-tokens` estimates (Cloudflare's Markdown for Agents stopped sending
them on 2026-10-09) and, as Cloudflare does, no `ETag`/`Last-Modified`, since conditional requests cannot
be honored for a converted body.

## What it does

- **Agent detection** from 22 operator-documented User-Agents (OpenAI, Anthropic, Perplexity,
  Google, Meta, Mistral, Apple, Amazon, ByteDance, Common Crawl). The catalog is generated from the
  same source as the TypeScript packages and both pass the same conformance vectors.
- **Optional IP verification** (`AgentVerifier` + `get_client_ip=client_ip_from_scope`): checks the
  client IP against the ranges each operator publishes → `verified`, `mismatch` (likely spoofed) or
  `declared`. Lists download in the background; failures fall back to `declared`.
- **Safe HTML → Markdown**: no regex over HTML, scripts/styles/comments/hidden content dropped,
  only `http(s)`/`mailto`/relative URLs kept, nesting capped at 512.
- **Metadata**: a YAML frontmatter (`title`, `description`, `image` from `<meta>`, with `name` taking
  precedence over Open Graph) whose values are JSON-quoted strings that cannot inject keys, and the
  page's valid JSON-LD appended as one fenced JSON value (an array when there are several blocks;
  malformed blocks and blocks with `javascript:`/`vbscript:`/`data:` strings are dropped; 64 KB total).
- **Resource limits**: at most `max_body_bytes` (2 MB) read per response; the app is told to stop
  (`http.disconnect`) once enough has been read; conversion runs in a worker thread; a conversion
  cache and per-client/global rate limits protect the CPU (rate-limited requests get the original HTML).
- **Telemetry**: `on_event` (sync or async) runs in the background with the path and agent — never
  the client IP or the query string. `IngestClient(write_key)` sends events to the AgentPulse API
  (`app.add_middleware(AgentPulseMiddleware, on_event=telemetry.on_event)`, then
  `await telemetry.shutdown()` in your lifespan). It keeps a bounded queue (500 events; the oldest
  are dropped and counted), sends batches every 10 s or at 50 events, and retries with backoff and
  `Retry-After`. Paths are redacted (emails, UUIDs, long numbers, hex and tokens) before they leave
  your server. Plain human requests are sent only as a counter.

## Requirements and security notes

- Python ≥ 3.10 **with the 2025 `html.parser` security fixes** (CVE-2025-6069, gh-102555,
  gh-135661). Older patch releases leak unclosed HTML comments as text and parse some inputs in
  quadratic time; the middleware checks the interpreter at startup and refuses to run on a
  vulnerable one (`UnsafeHtmlParserError`).
- asyncio servers (uvicorn, hypercorn).
- Behind a reverse proxy, let the server set the client address from trusted proxies only
  (`uvicorn --proxy-headers --forwarded-allow-ips=...`); never read `X-Forwarded-For` yourself.
- Shared caches/CDNs that ignore `Vary` could serve Markdown to browsers; configure them to vary on
  `Accept` or not to cache negotiated pages.

## License

MIT
