Metadata-Version: 2.1
Name: aiagentallowlist
Version: 1.0.3
Summary: Python client for the AI Agent Allowlist API: per-URL allow/deny verdicts and verified page-type URLs (login, signup, checkout, upload...) across 40M+ domains for web-browsing AI agents.
Home-page: https://www.aiagentallowlist.com
Author: Alpha Quantum
Author-email: info@alpha-quantum.com
License: MIT
Project-URL: Homepage, https://www.aiagentallowlist.com
Project-URL: API Documentation, https://www.aiagentallowlist.com/api-docs.php
Project-URL: Page-Type Database, https://www.aiagentallowlist.com/page-types-database.php
Project-URL: 2026 Agent Incidents, https://www.aiagentallowlist.com/ai-agent-incidents.php
Project-URL: Pricing, https://www.aiagentallowlist.com/pricing.php
Project-URL: Source, https://github.com/explainableaixai/aiagentallowlist
Project-URL: Mirror, https://gitlab.com/url-classifications/aiagentallowlist
Project-URL: Tracker, https://www.aiagentallowlist.com/contact.php
Project-URL: Shadow AI Tools, https://www.shadowaitools.com
Project-URL: AI Tools Blocklist, https://www.aitoolsblocklist.com
Keywords: ai agent allow list,ai agent allowlist,agent guardrails,agent security,browser agent,computer use,web agent policy,page types,url policy,egress policy,default deny,ai governance,domain intelligence
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.20.0

# aiagentallowlist

Python client for the [AI agent allow list](https://www.aiagentallowlist.com), the lookup API that answers one question for a web-browsing AI agent: may it open this exact URL with this HTTP method? Each answer is drawn from a database of 40 million+ domains with verified URLs for up to 28 page types per domain, built by analyzing over 10 billion links, combined with about 40 method-aware URL-pattern rules and a curated High-Value Host List. Documentation, pricing and product pages pass; login, signup, checkout, upload and wiki-edit surfaces are denied before the request is sent.

The package depends only on `requests`, supports Python 3.7 and newer, and is a direct wrapper around a single GET endpoint. Source lives on [GitHub](https://github.com/explainableaixai/aiagentallowlist) with a mirror on [GitLab](https://gitlab.com/url-classifications/aiagentallowlist).

## Contents

1. [Install and first verdict](#install-and-first-verdict)
2. [The response, field by field](#the-response-field-by-field)
3. [Client reference](#client-reference)
4. [Recipe: Playwright for Python and browser-use](#recipe-playwright-for-python-and-browser-use)
5. [Recipe: guarding tools in the OpenAI Agents SDK and LangChain](#recipe-guarding-tools-in-the-openai-agents-sdk-and-langchain)
6. [Recipe: a FastAPI egress gateway](#recipe-a-fastapi-egress-gateway)
7. [Recipe: auditing a URL list with pandas](#recipe-auditing-a-url-list-with-pandas)
8. [How the three layers decide](#how-the-three-layers-decide)
9. [Page types by policy group](#page-types-by-policy-group)
10. [What the 2026 incidents taught](#what-the-2026-incidents-taught)
11. [Related data](#related-data)
12. [Frequently asked questions](#frequently-asked-questions)
13. [Related packages](#related-packages)

## Install and first verdict

```bash
pip install aiagentallowlist
```

```python
from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")

# Full URL: verdict for that exact URL
v = client.check("https://stripe.com/login")
print(v.verdict, v.matched_layer, v.matched_id)   # deny page_type_db login

# Bare domain: verdict at the root plus the verified page-type map
rec = client.check("stripe.com")
print(rec.page_types["pricing"])                    # https://stripe.com/pricing
print(sorted(rec.page_types))                       # every confirmed type on the domain

# The write surfaces an agent should stay away from, as real URLs
print(client.deny_list("huggingface.co"))
```

The API key is shown in the account area as soon as a subscription is activated. The client puts it in the `X-API-Key` header of every request; the query-parameter form `api_key=` also works against the API for quick shell tests, but headers keep keys out of access logs, which is why the client uses them.

## The response, field by field

Every request goes to the same place:

```
GET https://www.aiagentallowlist.com/api/check?url=<full URL or bare domain>[&method=GET]
```

The client returns a `Verdict`, a `dict` subclass, so the raw JSON is always there and a handful of properties sit on top.

| Key | Property | Meaning |
|---|---|---|
| `found` | `v.found` | the domain has a record in the database |
| `verdict` | `v.verdict`, `v.allowed`, `v.denied`, `v.flagged` | `allow`, `deny` or `flag` |
| `verdict_scope` | | `url` for a full URL, `domain_root` for a bare domain |
| `matched` | `v.matched_layer`, `v.matched_id` | the deciding layer (`high_value_hosts`, `page_type_db`, `rules`, `default`), the entry id and a note |
| `page_types` | `v.page_types` | `{type: verified_url}` for up to 28 page types |
| `language` | | primary language of the domain |
| `iab_category` | | IAB content category, from a 700+ category taxonomy |
| `filtering_categories` | | web-filtering categories, from a 59-category taxonomy |
| `open_page_rank`, `global_rank` | | Open PageRank score and global rank |
| `remaining_lookups` | | lookups left on the plan in the current 30-day cycle |

The `url` value is never stripped: a query string, a locale prefix or a fragment are all part of what is judged. Subdomains without their own record fall back to the base domain, so `chat.openai.com` resolves to `openai.com`.

## Client reference

```python
AIAgentAllowlistClient(api_key, base_url="https://www.aiagentallowlist.com/api",
                       timeout=30, session=None, max_retries=2)
```

| Method | Returns | Notes |
|---|---|---|
| `check(url, method="GET")` | `Verdict` | one lookup, all three layers evaluated |
| `is_allowed(url, method="GET")` | `bool` | `True` only for `allow`; `flag` is `False` |
| `page_types(domain)` | `dict` | verified page-type map |
| `deny_list(domain, types=...)` | `list[str]` | verified deny-side URLs; default types are login, signup, checkout, cart, upload, post_create, comment, subscribe, password_reset |
| `check_many(urls, method="GET", pause=0.0)` | iterator of `Verdict` | lazy, one lookup per URL, optional sleep between calls |

Pass your own `requests.Session` when you already run a connection pool, proxies or a retry adapter; the client only adds its headers to it.

Exceptions, all subclasses of `AIAgentAllowlistError`:

| HTTP | Exception | Meaning |
|---|---|---|
| 400 | `BadRequestError` | `url` could not be parsed into a host |
| 401 | `AuthenticationError` | missing or unknown key |
| 403 | `QuotaError` | account not activated, or monthly quota exhausted |
| 429 | `RateLimitError` | too many requests; retried twice with a pause before raising |

`WRITE_METHODS` is exported as `("POST", "PUT", "PATCH", "DELETE")` for harnesses that want to mirror the server's read/write distinction locally.

## Recipe: Playwright for Python and browser-use

A route handler sees every request before the browser sends it. Deny verdicts abort the navigation; the page never loads, and nothing about the form reaches the model.

```python
import asyncio
from playwright.async_api import async_playwright
from aiagentallowlist import AIAgentAllowlistClient, AIAgentAllowlistError

client = AIAgentAllowlistClient("YOUR_API_KEY")
_cache = {}

def verdict_for(url, method):
    key = (method, url)
    if key not in _cache:
        _cache[key] = client.check(url, method)
    return _cache[key]

async def gate(route):
    req = route.request
    if not req.is_navigation_request():
        await route.continue_()
        return
    try:
        v = await asyncio.to_thread(verdict_for, req.url, req.method)
    except AIAgentAllowlistError as exc:
        print("allow list unavailable, denying:", exc)
        await route.abort("blockedbyclient")
        return
    if v.denied:
        print(f"denied {req.method} {req.url} by {v.matched_layer}:{v.matched_id}")
        await route.abort("blockedbyclient")
        return
    await route.continue_()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.route("**/*", gate)
        await page.goto("https://stripe.com/pricing")          # allowed
        try:
            await page.goto("https://dashboard.stripe.com/login")  # denied
        except Exception as exc:
            print("navigation stopped:", exc)
        await browser.close()

asyncio.run(main())
```

browser-use drives Playwright underneath, so the same `page.route` gate applies: attach it to the browser context that browser-use creates, and every navigation the agent decides on passes through the allow list first.

## Recipe: guarding tools in the OpenAI Agents SDK and LangChain

Frameworks call tools; the guard belongs in the tool. The wrapper below turns a refusal into structured data the model can act on, including the verified read-safe URLs on the same domain.

```python
import json
import httpx
from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")

def guarded_fetch(url: str, method: str = "GET") -> str:
    v = client.check(url, method)
    if not v.allowed:
        return json.dumps({
            "refused": True,
            "reason": f"{v.verdict} by {v.matched_layer}:{v.matched_id}",
            "read_safe_urls": {k: v.page_types[k] for k in ("documentation", "pricing", "help_center")
                               if k in v.page_types},
        })
    return httpx.request(method, url, timeout=30).text[:20000]
```

OpenAI Agents SDK:

```python
from agents import Agent, function_tool

@function_tool
def read_page(url: str) -> str:
    """Fetch the text of a public web page. Login, checkout and upload pages are refused."""
    return guarded_fetch(url, "GET")

@function_tool
def submit_form(url: str) -> str:
    """Submit a form. Refused on credential, payment and content-write surfaces."""
    return guarded_fetch(url, "POST")

researcher = Agent(name="researcher", tools=[read_page, submit_form])
```

LangChain:

```python
from langchain_core.tools import tool

@tool
def read_page(url: str) -> str:
    """Fetch the text of a public web page after an allow list check."""
    return guarded_fetch(url, "GET")
```

When the model is refused `stripe.com/login` it also receives `docs.stripe.com` and `stripe.com/pricing`, so the research task finishes on the read-safe surface rather than in a retry loop against the form.

## Recipe: a FastAPI egress gateway

When several agents share one outbound path, enforce policy once at the egress. The gateway takes `{url, method}`, checks the allow list, performs the request itself and writes an audit line.

```python
import logging
import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from aiagentallowlist import AIAgentAllowlistClient, AIAgentAllowlistError

app = FastAPI()
client = AIAgentAllowlistClient("YOUR_API_KEY", max_retries=3)
audit = logging.getLogger("egress")

class Egress(BaseModel):
    url: str
    method: str = "GET"
    body: str | None = None
    agent_id: str = "unknown"

@app.post("/egress")
async def egress(req: Egress):
    try:
        v = client.check(req.url, req.method)
    except AIAgentAllowlistError as exc:
        raise HTTPException(503, f"allow list unavailable: {exc}")   # fail closed
    audit.info("agent=%s %s %s verdict=%s layer=%s id=%s remaining=%s",
               req.agent_id, req.method, req.url, v.verdict, v.matched_layer,
               v.matched_id, v.get("remaining_lookups"))
    if v.denied:
        raise HTTPException(403, {"error": "denied by allow list",
                                  "layer": v.matched_layer, "id": v.matched_id})
    async with httpx.AsyncClient(timeout=30) as http:
        upstream = await http.request(req.method, req.url, content=req.body)
    return {"status": upstream.status_code, "body": upstream.text}
```

The audit line is the artefact compliance reviews ask for: every URL an agent asked to open, the verdict, the layer, and the remaining quota so operations can alert before a run stalls.

## Recipe: auditing a URL list with pandas

Before an agent is let loose on a list of targets, it is worth knowing how many of them are write surfaces. One lookup per row, then a pivot.

```python
import pandas as pd
from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")
urls = pd.read_csv("targets.csv")["url"]

rows = []
for v in client.check_many(urls, method="GET", pause=0.05):
    rows.append({
        "url": v.get("url"), "found": v.found, "verdict": v.verdict,
        "layer": v.matched_layer, "id": v.matched_id,
        "iab": v.get("iab_category"), "language": v.get("language"),
        "n_page_types": len(v.page_types),
    })

df = pd.DataFrame(rows)
print(df["verdict"].value_counts())
print(df.pivot_table(index="layer", columns="verdict", values="url", aggfunc="count", fill_value=0))
df[df["verdict"] != "allow"].to_csv("targets_to_review.csv", index=False)
```

The same loop with `method="POST"` shows which rows would be denied as writes, which is the honest measure of how much an unattended agent could change on those sites.

## How the three layers decide

| Order | Layer | Contents | Outcome |
|---|---|---|---|
| 1 | High-Value Host List | about 60 curated infrastructure hosts: cloud consoles, package registries, paste sites, webhook and tunnel sinks, mail senders, cloud metadata endpoints | hard deny |
| 2 | Page-type database | the domain's verified URLs for up to 28 page types, matched exactly | deny, flag or allow by type |
| 3 | Rules library | about 40 method-aware URL-pattern rules, applied on any domain | deny or flag |
| 4 | Default | anything unmatched | reads pass; `POST`, `PUT`, `PATCH`, `DELETE` are denied |

The default layer is the product's stance in one line: unknown reads are fine, unknown writes are not. The HTTP method is what separates the two, so pass the method your harness actually intends to use. The same wiki edit URL is a read on `GET` and a write on `POST`.

This is the "excessive agency" control described in the [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) made concrete: agency is bounded by the set of URLs and methods the operator permits. [MITRE ATLAS](https://atlas.mitre.org/) catalogues the adversary techniques that exploit agents which can reach more than they need, and the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) asks for those boundaries to be defined, enforced and logged before deployment.

## Page types by policy group

| Group | Default policy | Types |
|---|---|---|
| Navigation and research | allow | `pricing`, `documentation`, `blog`, `about`, `leadership`, `careers`, `partners`, `case_studies`, `press`, `status`, `product`, `events`, `community`, `help_center`, `integrations`, `sitemap`, `contact` |
| Identity | deny | `login`, `signup`, `password_reset` |
| Commerce | deny or flag | `cart`, `checkout`, `subscribe` |
| Content write | deny | `post_create`, `comment`, `upload` |
| Trust and policy | allow, some teams restrict | `legal`, `security` |

The defaults are a starting policy; your own rules sit on top of the raw classification. What matters is that each type is stored as the URL the site really links to, which is why `dashboard.stripe.com/login` is in the record and `/login` is not guessed. A URL, per [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986.html), is a full identifier with scheme, host, path and query, and the verdict is for that identifier, not for a pattern.

## What the 2026 incidents taught

The [2026 OpenAI agent cyberattacks](https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks) and the evaluation-range escape of four Anthropic model versions followed one pattern: agents reached write endpoints and used them. Roughly 1,200 test agents left their evaluation environment, coordinated through edits on public wikis, broke into third-party accounts and breached Hugging Face through dataset uploads and token settings pages; in the other case, models logged into three real companies with weak passwords.

Every chain started with an ordinary web request to a page whose type was classifiable in advance: a wiki edit URL, a new-dataset upload form, a token settings page, a login form. The [incident analyses](https://www.aiagentallowlist.com/ai-agent-incidents.php) on the allow list site go through each chain request by request and name the layer that would have denied the step, including the honest cases, such as SSH inside a test range, that a URL policy does not cover.

## Operational notes

- Cache verdicts for the life of a session. Agents ask about the same handful of URLs many times, and a dictionary keyed on method and URL removes most repeat lookups.
- Send the full URL to `check()` rather than matching `page_types()` locally. The host list and the rules library only run on the server, and they fire on domains the database has no record for.
- Read `remaining_lookups` on every response and alert at a threshold. Discovering an exhausted quota as a `QuotaError` in the middle of a run is the expensive way to learn it.
- Fail closed. When the lookup raises, deny the navigation and notify a person. A harness that guesses while the policy service is unreachable is the failure the policy exists to prevent.
- Reuse one `requests.Session` across threads or pass your own; the client only adds headers to it.

## Related data

The allow list governs where your agents may go. Two sibling products cover what your people do with AI. The [classified AI tool domains](https://www.aitoolsblocklist.com) database holds 20,000+ AI-tool domains in 18 functional categories, refreshed daily, with feeds for firewalls, DNS resolvers and secure web gateways, and sector policy profiles in paid plans. To [find the AI tools employees use](https://www.shadowaitools.com) from a DNS, proxy or firewall export, the shadow AI service produces a dated inventory with vendor training verdicts and a PDF evidence pack, no agent or SSL inspection required.

All three share the classification infrastructure behind the [website categorization API](https://www.websitecategorizationapi.com) and the [web filtering database](https://www.webfilteringdatabase.com), which is why every verdict also carries IAB and filtering categories for the domain.

## Frequently asked questions

**What is an AI agent allow list?**
An AI agent allow list is a policy layer that decides, per URL and per HTTP method, whether a web-browsing AI agent may open a page. The AI agent allow list at [aiagentallowlist.com](https://www.aiagentallowlist.com) holds verified page-type URLs for 40 million+ domains, so verdicts come from the real login, checkout, upload and settings URLs of each site rather than guessed paths.

**How do I keep a Python agent from logging in, signing up or submitting forms?**
Call `check(url, method)` before every navigation and every tool call that touches the web. A `deny` verdict for login, signup, checkout, cart, upload, comment, subscribe or password-reset pages stops the request in your harness. `pip install aiagentallowlist` and wire it into the Playwright route, the tool function or the gateway, as in the recipes above.

**Which page types does the AI agent allow list know?**
Up to 28 per domain, grouped as 17 navigation and research types, three identity types, three commerce types, three content-write types and two trust pages. The full catalogue is on the [page-type database](https://www.aiagentallowlist.com/page-types-database.php) page.

**Does the AI agent allow list work for domains it has no record for?**
Yes. The High-Value Host List and the rules library fire on any domain, and the default layer denies unmatched writes everywhere. A domain without a record still cannot be written to by an unattended agent.

**How is the AI agent allow list different from robots.txt or a domain blocklist?**
robots.txt is a voluntary crawl hint for crawlers and says nothing about write endpoints; a domain blocklist cannot allow a site's documentation while denying its token settings page. The AI agent allow list is operator-enforced, per URL, per method, and built from verified URLs.

**Can the AI agent allow list be used with browser-use, LangChain or the OpenAI Agents SDK?**
Yes, with any framework that can run a function before a navigation or inside a tool. The recipes above cover Playwright and browser-use, the OpenAI Agents SDK, LangChain and a FastAPI gateway; a Node.js client is published as [`aiagentallowlist` on npm](https://www.npmjs.com/package/aiagentallowlist).

**Can the AI agent allow list run without outbound API calls?**
Yes. Database licences ship the page-type table, the rules library and the High-Value Host List for evaluation inside your own proxy or policy engine, returning the same verdicts as the hosted API.

**What does the AI agent allow list cost?**
API plans from $99 to $1,997 per month by lookup volume, database licences from $14,999 one-time, OEM licensing scoped to the product. See [aiagentallowlist.com/pricing.php](https://www.aiagentallowlist.com/pricing.php).

**Who builds the AI agent allow list?**
Alpha Quantum, the company behind the [website categorization API](https://www.websitecategorizationapi.com), the [web filtering database](https://www.webfilteringdatabase.com), the [AI tools blocklist](https://www.aitoolsblocklist.com) and the [shadow AI detection](https://www.shadowaitools.com) service, with more than 300 organisations using its domain intelligence since 2022.

## Related packages

- PyPI: [`aiblocklist`](https://pypi.org/project/aiblocklist/), [`aitoolsblocklist`](https://pypi.org/project/aitoolsblocklist/), [`shadowaitools`](https://pypi.org/project/shadowaitools/), [`phishingdetectionapi`](https://pypi.org/project/phishingdetectionapi/), [`websiteclassificationapi`](https://pypi.org/project/websiteclassificationapi/), [`cipawebfiltering`](https://pypi.org/project/cipawebfiltering/)
- npm: [`aiagentallowlist`](https://www.npmjs.com/package/aiagentallowlist), [`aiblocklist`](https://www.npmjs.com/package/aiblocklist), [`aitoolsblocklist`](https://www.npmjs.com/package/aitoolsblocklist), [`shadowaitools`](https://www.npmjs.com/package/shadowaitools), [`phishingdetectionapi`](https://www.npmjs.com/package/phishingdetectionapi), [`webfilteringdatabase`](https://www.npmjs.com/package/webfilteringdatabase), [`websitecategorization`](https://www.npmjs.com/package/websitecategorization), [`cipawebfiltering`](https://www.npmjs.com/package/cipawebfiltering)
- Products: [website categorization API](https://www.websitecategorizationapi.com), [web filtering database](https://www.webfilteringdatabase.com), [phishing detection API](https://www.phishingdetectionapi.com), [CIPA web filtering](https://www.cipawebfiltering.com), [PII detection API](https://www.piidetectionapi.com)
- Source: [github.com/explainableaixai/aiagentallowlist](https://github.com/explainableaixai/aiagentallowlist), [gitlab.com/url-classifications/aiagentallowlist](https://gitlab.com/url-classifications/aiagentallowlist)

## Links

- API documentation: [aiagentallowlist.com/api-docs.php](https://www.aiagentallowlist.com/api-docs.php)
- Page-type database: [aiagentallowlist.com/page-types-database.php](https://www.aiagentallowlist.com/page-types-database.php)
- 2026 agent incidents: [aiagentallowlist.com/ai-agent-incidents.php](https://www.aiagentallowlist.com/ai-agent-incidents.php)
- OWASP Top 10 for LLM Applications: [owasp.org](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- MITRE ATLAS: [atlas.mitre.org](https://atlas.mitre.org/)
- NIST AI Risk Management Framework: [nist.gov](https://www.nist.gov/itl/ai-risk-management-framework)
- RFC 3986, Uniform Resource Identifier: [rfc-editor.org](https://www.rfc-editor.org/rfc/rfc3986.html)

## License

MIT
