Metadata-Version: 2.4
Name: readystack-robots-txt-ai-crawler-audit
Version: 0.1.8
Summary: Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI-search citation, or user fetch - and which lines silently do nothing. (bundled Node.js runtime)
Author-email: ReadyStack <support@getreadystack.com>
License: Proprietary - see LICENSE.txt
Project-URL: Homepage, https://getreadystack.com/tools/robots-txt-ai-crawler-audit?ref=pypi
Project-URL: Documentation, https://getreadystack.com/tools/robots-txt-ai-crawler-audit?ref=pypi
Project-URL: Support, https://getreadystack.com/support
Keywords: robots.txt,AI crawler,GPTBot,Google-Extended,SEO
Classifier: Programming Language :: Python :: 3
Classifier: Environment :: Console
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: nodejs-wheel-binaries>=20
Dynamic: license-file

# AI Crawler Rules - robots.txt Audit for AI Search

> Runs the same checker as the npm package `@readystack/robots-txt-ai-crawler-audit` on a Node.js runtime that pip installs for you (`nodejs-wheel-binaries`) - no system Node.js needed. Your files are checked locally.

![robots.txt AI Crawler Audit - GPTBot, ClaudeBot, Google-Extended — finds the line](https://getreadystack.com/img/promo/robots-txt-ai-crawler-audit_demo.gif)

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI-search citation, or user fetch - and which lines silently do nothing.

## Install

```
robots-txt-ai-crawler-audit file
```

Node 18+. The same 19 rules as the VS Code extension, from a terminal or CI.

## Free

- Check the open robots.txt against every rule - full results, nothing withheld; Check only the lines you highlight; See every rule and what each AI token controls; Reopen the last report
- `--rules` lists every rule

## With a licence ($29 once)

- Scan every robots.txt in the workspace; Export the report as CSV, JSON or HTML; CI output that fails the build on errors; Add your own agency policy tokens; Re-check automatically on every save

```
@readystack/robots-txt-ai-crawler-audit --dir ./templates --report html --out report.html
```

A freelance technical SEO consultant bills roughly $100-150 an hour, and an AI-crawler robots.txt review is a one-to-two hour job.

## Use from an AI agent (MCP)

Claude Code · Cursor · Windsurf · any MCP client - add to your MCP config:

```json
{ "mcpServers": { "robots-txt-ai-crawler-audit": { "command": "npx", "args": ["-y", "@readystack/robots-txt-ai-crawler-audit", "--mcp"] } } }
```

Tools: `check_text` and `check_file` (free) · `check_dir` (licence). The agent gets every finding with the line number.

## Use in CI

```yaml
- name: AI Crawler Rules - robots.txt Audit for AI Search
  run: npx -y @readystack/robots-txt-ai-crawler-audit --dir . --ci
```

(container: `docker run --rm -v "$PWD:/work" getreadystack/robots-txt-ai-crawler-audit --dir /work --ci`)

The folder sweep, reports and CI mode need one licence — one payment, no subscription. Set `READYSTACK_LICENSE=<key>` or run `--license <key>` once.

[Get a licence](https://getreadystack.com/api/buy/cl/polar_cl_I8R8AzowzmASHy45ujvewcvtcl4wBBfC2Fe7r3fZwpr)


<!-- ai crawler rules for robots txt -->


## Install (PyPI)

```
pip install readystack-robots-txt-ai-crawler-audit
robots-txt-ai-crawler-audit --help
```
