Guides / AI-bot policy

An AI-bot policy in robots.txt

Worth +12 points · makes your stance on AI crawlers intentional

AI companies run their own crawlers, separate from the classic search bots. If your robots.txt says nothing about them, your policy is accidental. Stating it explicitly — whether you allow them (to be cited in answers) or block them (to keep content out of training/answers) — is the AI-ready move. To be found by ChatGPT, Claude and Perplexity, you generally want to allow them.

The major AI crawlers

User-agentWho
GPTBot, OAI-SearchBot, ChatGPT-UserOpenAI / ChatGPT
ClaudeBot, anthropic-aiAnthropic / Claude
Google-ExtendedGoogle Gemini / AI Overviews
PerplexityBotPerplexity
CCBotCommon Crawl (feeds many models)

Copy-paste: allow AI crawlers

Add this to your robots.txt at https://yourdomain.com/robots.txt:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

To block a crawler instead, replace its Allow: / with Disallow: /. Either way, naming them is what makes the policy intentional.

Updated your robots.txt?
Re-grade your site →