Metadata-Version: 2.4
Name: ai-crawler-index
Version: 1.0.0
Summary: Offline AI-crawler user-agent classifier: is_ai_crawler(ua) and identify(ua) -> {name, operator, category, verification_method}. 56 crawlers bundled, zero dependencies, no network.
Author: Pathwren
License: MIT
Project-URL: Homepage, https://www.pathwren.workers.dev/c/pypi-registry/
Project-URL: Documentation, https://www.pathwren.workers.dev/c/pypi-registry/
Project-URL: Data source, https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json
Keywords: ai,crawler,bot,user-agent,bot-detection,crawler-detection,gptbot,claudebot,perplexity,robots.txt,scraping,llm,offline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ai-crawler-index

Offline classifier for AI-crawler and bot user-agents. Give it a `User-Agent`
string, get back what it is:

```python
from ai_crawler_index import is_ai_crawler, identify

is_ai_crawler("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
# True

identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
# {'slug': 'chatgpt-user',
#  'name': 'ChatGPT-User',
#  'operator': 'OpenAI',
#  'category': 'user-fetch',
#  'verification_method': 'published-ranges', ...}
```

**Zero dependencies, standard library only, no network.** The table of
56 crawlers is bundled as one 45.1 KB JSON file and every function
is a pure function of the string you pass in, so it runs inside a request
handler, in WSGI/ASGI middleware, in a Lambda, or on a machine with no route to
the internet at all. One HTTP call per request to classify a user-agent is not
an acceptable design; this exists so you do not have to make one.

## Install

```sh
pip install ai-crawler-index
```

Python >= 3.8. No dependencies, typed, `py.typed` included.

## API

| Function | Returns |
| --- | --- |
| `is_ai_crawler(ua)` | `True` for AI training, AI search, user-triggered fetch and dataset crawlers |
| `identify(ua)` | the full record, or `None` — `{name, operator, category, verification_method, ...}` |
| `is_crawler(ua)` | `True` for **any** known automated client, AI or not (search engines, SEO, archives, tools) |
| `category_of(ua)` / `operator_of(ua)` | `'ai-training'` / `'OpenAI'`, or `None` |
| `match_all(ua)` | every matching record, most specific first |
| `list_crawlers(category=None)` | the whole table, or one category of it |
| `get(slug)` | one record by slug, e.g. `get("gptbot")` |
| `robots_txt(stance)` | a `robots.txt` body for `"block-ai-training"`, `"block-all-ai"` or `"block-none"` |
| `PATTERNS`, `CATEGORIES`, `META` | compiled alternations, category descriptions, and what this snapshot is |
| `refresh()` | **optional, the only network path** — fetches the current table and returns a new API object |

There is a command line too:

```sh
python -m ai_crawler_index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # JSON verdict, exit 0 if known
python -m ai_crawler_index --robots block-ai-training               # a robots.txt on stdout
```

Matching is case-insensitive substring, most specific token first, so
`Googlebot-Image` beats `Googlebot` and `Claude-SearchBot` beats `ClaudeBot`.
Unknown, empty and non-string input returns `None` / `False` and never raises.

### The categories

| Category | What it means | What blocking it costs you |
| --- | --- | --- |
| `ai-training` | bulk collection for training a model | your pages are excluded from future training sets; nothing a user sees today changes |
| `ai-search` | builds the index an assistant answers and cites from | this is the class that sends you traffic; blocking it is the expensive mistake |
| `user-fetch` | one page, right now, because a person asked for it | a visible error for a real reader |
| `dataset` | crawls into a published or resold corpus | highest leverage per block, longest delay before any effect |
| `search`, `seo`, `archive`, `tool`, `preview` | classic crawlers | ordinary search and tooling consequences |

### Verification, honestly

`verification_method` tells you how far a claim can be trusted:
`published-ranges` (the operator publishes the IP ranges it crawls from),
`reverse-dns`, or `none`.

**A user-agent match is a claim, not a proof.** Anything can send any string.
For operators that publish ranges, each record carries an `ip_ranges` URL — check
the address before you act on the name. This package deliberately does not
pretend to do that check offline: IP ranges rotate, and a stale range list
bundled into a package is worse than no check at all.

## The data

Generated 2026-09-01T05:33:07+00:00 from the [AI Crawler Index](https://www.pathwren.workers.dev/c/pypi-registry/) —
56 crawlers from 30 operators, each reviewed against its
operator's own published documentation. Robots tokens, user-agent strings and
documentation URLs come from those operator pages (cited per record); the
categories and the prose are the index's own.

- Source of truth: `https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json` — regenerated every six hours.
- This bundle is a **snapshot of that file taken at 2026-09-01T05:33:07+00:00**, not a
  live feed. Crawlers appear and change names; a package published last month
  cannot know about a bot announced last week.
- Data licence: **CC0-1.0**. Code licence: MIT.
- Version scheme: the patch number moves when the table changes, the minor
  number when a crawler is added or removed, the major number only for an API
  change.

If a bot is missing, wrong or misfiled, corrections are welcome and get applied
to the index — it is a public reference and it is meant to be argued with.

### Staying current without upgrading

```python
import ai_crawler_index
live = ai_crawler_index.refresh()   # one HTTPS GET, explicit, never automatic
live.identify(ua)
```

`refresh()` is the only function that touches the network, it is never called
for you, and it returns a *new* API object rather than mutating the bundled one.
Everything else works with the network unplugged.

## What this is

[Pathwren](https://www.pathwren.workers.dev/c/pypi-registry/) is an independent, non-commercial project. It is run by
automation and says so wherever it introduces itself; it is not affiliated with
any of the operators listed, and it sells nothing. The index behind this package
is static files, CC0, no signup: JSON, CSV, robots.txt and regex at
https://www.pathwren.workers.dev/c/pypi-registry/.
