robots.txt for AI in 2026: block, allow, or something in between

The core insight most people miss

Every major AI company runs multiple bots, and they do different jobs. Blocking one doesn't block them all. Allowing one doesn't allow them all. If you set up AI-crawler rules without knowing which bot does what, you'll either accidentally block yourself from valuable citations, or accidentally let training crawlers scrape you when you didn't want them to.

Here's the map for 2026.

OpenAI (ChatGPT + related)

• GPTBot — the AI training crawler. Reads your site to potentially include content in future GPT training data. Blocking this doesn't affect ChatGPT answering questions about you today; it affects future model updates.
• OAI-SearchBot — indexes your site for the ChatGPT Search feature. Blocking this hides you from ChatGPT's search results.
• ChatGPT-User — fetches pages in real time when a user asks ChatGPT about a specific URL. Blocking this makes ChatGPT unable to browse to your site during conversations.

Most sites should allow all three unless you have a specific reason not to (paid content, legal restrictions).

Anthropic (Claude)

• ClaudeBot — AI training crawler.
• Claude-User — real-time browsing when a Claude user asks about a URL.
• Claude-SearchBot — search index crawler.

Same structure as OpenAI. Same recommendation — allow all three unless you have a policy reason not to.

Google AI

Google's AI stack is more complicated because Google has both search and AI products.

• Googlebot — regular search index. If you block this, you disappear from Google Search entirely. Almost nobody wants this.
• Google-Extended — Gemini and AI Overview training. Separate from Googlebot. You can allow Googlebot and block Google-Extended if you want to appear in search but not train Gemini.
• GoogleOther — internal Google product crawler.

Recommendation: allow Googlebot (obviously) and allow Google-Extended unless you're publishing content you don't want in AI training corpora (news orgs, licensed content).

Perplexity

• PerplexityBot — index crawler.
• Perplexity-User — live browsing.

Perplexity always cites sources in its answers, so being in its index directly drives referral traffic. Allow both.

The 40+ others

Beyond the big four, there are dozens of smaller AI crawlers: Grok (xAI), Copilot (Microsoft), Meta AI (meta-externalagent), DeepSeek, Mistral, Cohere, CCBot (Common Crawl), DuckAssistBot, Bytespider (ByteDance/TikTok), Diffbot, AI2Bot (Allen AI), Kagi-Bot, PhindBot, YouBot, Applebot-Extended (Apple Intelligence), Amazonbot, PetalBot (Huawei), and more.

Trying to hand-maintain per-bot rules for all of these is a losing battle. The bots come and go faster than you can update your robots.txt. Better strategies:

1. Allow everything if you're a public-facing business that wants maximum AI discoverability. Wildcard Allow: /.
2. Block training-only bots if you want AI systems to answer questions about you but not train on your content. This is what the new Content-Signaldirective is for.
3. Block everything if your content is licensed, paywalled, or otherwise sensitive. Wildcard Disallow: /.

The new Content-Signal directive

Introduced by Cloudflare and now backed by a growing group of AI companies, Content-Signal lets you express a nuanced preference in one line:

User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes

This says: don't use my content to train models, but DO include me in AI search results and DO fetch me as context when a user asks a question. It's the middle path most businesses want.

A recommended robots.txt for most sites

# Standard Googlebot access
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes

# Explicitly allow the major AI assistants
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Point AI systems at your sitemap and llms.txt
Sitemap: https://yourdomain.com/sitemap.xml

This is a balanced default. It allows search-based AI answers (which drive real referral traffic) while declining training use (which doesn't).

How to check whether your rules actually work

1. Paste your URL into our free robots.txt Validator. It parses your file line-by-line and flags syntax errors, orphaned rules, empty user-agent blocks, and unknown directives.
2. Run our AI Readiness Checker — it evaluates your robots.txt against 50+ AI crawler user-agents and gives you a pass/fail per bot.
3. For deeper analysis, our robots.txt Generator lets you pick a preset (Allow all, Search-only, Recommended balance, Block all) and generates a compliant merged file with your existing robots.txt content preserved.

Common mistakes

• Blocking Googlebot by accident. A User-agent: * Disallow: /block disappears you from Google Search. Very common on staging sites accidentally promoted to production.
• Assuming Google-Extended = Googlebot. They're separate. You can allow one and block the other.
• Using Noindex in robots.txt. Google stopped honoring this in 2019. Use a meta robots tag or X-Robots-Tag header instead.
• Not updating for new bots. New AI crawlers emerge every quarter. Recheck your file every 6-12 months.

Bottom line

For 90% of businesses, the right robots.txt for AI is: allow the major search + assistant bots, use the Content-Signal directive to opt out of training if you care about that, and use a tool to keep the file up to date as the ecosystem evolves.

Start with our free robots.txt Generator — pick a preset, download the merged file, upload to your site root. Ten minutes end-to-end.