Metadata-Version: 2.4
Name: ai-crawler-logs
Version: 1.0.0
Summary: Access-log triage for AI crawlers: identify every line, attribute it to its operator, and generate a robots.txt or WAF rule for the traffic you actually received. 56 crawlers bundled, six log formats auto-detected, zero dependencies.
Author: Pathwren
License: MIT
Project-URL: Homepage, https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/
Project-URL: Documentation, https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/
Project-URL: Data source, https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/data.json
Project-URL: Companion package, https://pypi.org/project/ai-crawler-index/
Keywords: logs,access-log,log-analysis,nginx,apache,crawler,bot-detection,ai,gptbot,user-agent,robots.txt,waf,offline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: Log Analysis
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Security
Classifier: Topic :: System :: Systems Administration
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: verify
Requires-Dist: ai-crawler-verify>=1.0; extra == "verify"
Dynamic: license-file

# ai-crawler-logs

**Who was actually in your access log?** Paste the log in whatever shape your
stack emits it. Every crawler gets identified and attributed to its operator,
rolled up by category, and turned into a robots.txt or an edge rule for the
traffic you really received.

```python
from ai_crawler_logs import triage

report = triage(open("access.log").read())

report["summary"]["by_operator"]   # {'OpenAI': 412, 'Anthropic': 88, 'Google': 61, ...}
report["summary"]["by_category"]   # {'ai-training': 500, 'ai-search': 91, ...}
report["summary"]["ai_share"]      # 0.4231
report["unmatched"]                # the strings nothing in the table explains
```

## Install

```sh
pip install ai-crawler-logs
```

Python >= 3.8. **Zero dependencies, standard library only.** No network at
import, no DNS ever: 56 crawler records and 8 ready-made
robots.txt stances ship inside the wheel (68.1 KB), snapshot taken
2026-09-01.

## It reads the log you have

Formats are detected **per line**, so a mixed paste works and every row reports
how it was read:

| format | example |
| --- | --- |
| combined / common | `1.2.3.4 - - [01/Sep/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 512 "-" "GPTBot/1.2"` |
| JSON lines | `{"remote_addr":"1.2.3.4","http_user_agent":"GPTBot/1.2","status":200}` |
| Cloudflare logpush | `{"ClientIP":"1.2.3.4","ClientRequestUserAgent":"GPTBot/1.2"}` |
| `uniq -c` frequency table | `   412 Mozilla/5.0 (compatible; GPTBot/1.2; ...)` |
| tab or comma separated | `412\tGPTBot/1.2` |
| `<ip> <ua>` pairs | `1.2.3.4 GPTBot/1.2` |
| a bare user-agent per line | `GPTBot/1.2` |

The everyday path is one shell line and no log parsing at all:

```sh
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | ai-crawler-logs -
```

## Command line

```sh
ai-crawler-logs access.log                    # who was in it
ai-crawler-logs access.log --ai-only          # only training/search/fetch/dataset crawlers
ai-crawler-logs access.log --json             # the whole report, one row per line
ai-crawler-logs access.log --robots block-ai-training
ai-crawler-logs access.log --waf nginx        # apache | caddy | cloudflare | haproxy
ai-crawler-logs access.log --verify           # needs ai-crawler-verify installed
ai-crawler-logs --stances                     # the 8 ready-made stances
```

```
$ ai-crawler-logs access.log
1284 request(s), 743 identified (41.2% AI), 96 distinct address(es)
formats: combined=1284

crawler                        operator          category        hits
GPTBot                         OpenAI            ai-training      412
ClaudeBot                      Anthropic         ai-training       88
PerplexityBot                  Perplexity        ai-search         61
...

not in the table (23 distinct):
   1204  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537...  browser-shaped string
     31  python-requests/2.31.0                                        python-requests
```

## The output you can paste

```sh
$ ai-crawler-logs access.log --robots block-ai-training
# Generated by ai-crawler-logs 1.0.0 from an access log.
# Stance: block-ai-training — ...
# Only the 6 crawler(s) that appeared in the log are named.

User-agent: GPTBot
Disallow: /
...
```

**Only what was actually there.** A robots.txt naming 56 tokens you
have never received is a file nobody will ever maintain; pass `--all` if you
want the whole stance anyway.

For the fetchers that will not be stopped by robots.txt at all, `--waf` writes
the edge rule instead:

```sh
$ ai-crawler-logs access.log --waf nginx
map $http_user_agent $ai_crawler {
    default 0;
    ~*(GPTBot|ClaudeBot|PerplexityBot) 1;
}
if ($ai_crawler) { return 403; }
```

## A user-agent is a claim

Everything above is a claim matched against a table. Nothing in it is proof that
the client is who it says — the string is free to type, and the busiest strings
in most logs are forged. With
[`ai-crawler-verify`](https://pypi.org/project/ai-crawler-verify/) installed the
rows that carried an address get checked against the operator's own published
ranges:

```sh
pip install ai-crawler-logs[verify]
```

```
CLAIM CONTRADICTED BY ADDRESS (3):
  line 88    203.0.113.9      claims GPTBot — address is in no prefix OpenAI publishes
```

That list is the only thing here that rests on evidence, and it can still be
wrong two ways: a mirrored prefix list is up to six hours stale, and some
operators route through addresses they do not publish. It is a reason to look,
not a verdict about a person.

## Where the data comes from

The [AI Crawler Index](https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/) — an independent, non-commercial public
reference. Every crawler record is checked against its operator's own published
documentation. `refresh()` fetches today's table from
[https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/data.json](https://www.pathwren.workers.dev/c/pypi-registry/ai-crawler-logs/data.json) and caches it six hours; it is the only network
call in the package and you have to name it.

## Companion packages

- [`ai-crawler-index`](https://pypi.org/project/ai-crawler-index/) — what does
  this user-agent claim to be?
- [`ai-crawler-verify`](https://pypi.org/project/ai-crawler-verify/) — is that
  claim true?
- [`ai-crawler-robots`](https://pypi.org/project/ai-crawler-robots/) — does your
  robots.txt block the crawlers you think it blocks?

## Licence

Code MIT. Bundled data CC0-1.0. Independent and non-commercial; not affiliated
with, endorsed by or speaking for any crawler operator named in the table.
