Robots.txt for AI Crawlers
This is a configuration reference, not a definitions page — see AI Crawlers Explained for what each crawler actually does. Below is a comprehensive robots.txt template covering AI search/answer engines, the search engines that feed them, training crawlers, and control tokens, organized so you can allow or block each category deliberately rather than by accident.
Crawler names and policies change. Cross-check each token against the source links at the bottom of this page before deploying, especially for a production site.
A Complete AI-Crawler robots.txt Template
# ==========================================
# AI SEARCH & ANSWER ENGINES
# ==========================================
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
# ==========================================
# SEARCH ENGINES FEEDING AI SURFACES
# ==========================================
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /
User-agent: Applebot
Allow: /
# ==========================================
# AI TRAINING / DATA COLLECTION
# (independent of search visibility above)
# ==========================================
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: CCBot
Allow: /
User-agent: Bytespider
Allow: /
User-agent: Amazonbot
Allow: /
# ==========================================
# DATA-USE CONTROL TOKENS
# (not standalone crawlers — read alongside
# Googlebot/Applebot's own crawl)
# ==========================================
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThis template allows everything. Change any Allow to Disallow based on your own position on training versus search visibility for that specific crawler — there's no universally "correct" configuration.
Why the Categories Matter
| Category | What blocking it does |
|---|---|
| Search / answer engine | Removes eligibility for citations on that specific platform |
| Search engine feeding AI | Removes the site from that engine's regular index — and anything built on top of it |
| Training crawler | Opts out of model training only — no effect on search/citation eligibility |
| Control token | Governs data-use permission for an existing crawl — not a fetch itself |
Apple is the clearest documented example of the last category: Apple states Applebot-Extended does not crawl webpages at all — it only controls whether content already fetched by the regular Applebot may be used to train Apple's foundation models. Confusing a control token for an independent crawler is one of the most common configuration errors. Established
Configuration Isn't the Same as Access
A syntactically correct robots.txt only states intent. A CDN, WAF, or bot-management rule can still block the actual HTTP request before your server ever sees it, and rendering behavior (whether a crawler executes JavaScript) determines whether an allowed crawler can actually read your content once it arrives. See What Is AI Crawlability? for the full three-layer breakdown, and verify live with each crawler's real user-agent rather than trusting the file alone.
Common Mistakes
- Inheriting a blanket "block all AI bots" rule from a starter template without realizing it also blocks search/citation-relevant crawlers, not just training ones.
- Treating a control token as a real crawler and expecting blocking it to stop content collection — it doesn't fetch anything itself.
- Never revisiting the file. New crawlers and tokens are introduced regularly; a robots.txt written a year ago is likely incomplete.
Where This Fits in NOXEL
NOXEL's AI Crawler Accessibility check tests this exact registry live over HTTP for every scan, catching both misconfigured robots.txt and independent firewall-layer blocks the file alone can't reveal.
Related Concepts
Part of The Complete Guide to AI Search Visibility (2026)
Sources
- OpenAI Developer Docs. "Overview of OpenAI Crawlers." developers.openai.com
- Anthropic Help Center. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" support.claude.com
- Perplexity. "Bots / Perplexity Crawlers." docs.perplexity.ai
- Google Search Central. "Google Crawlers (User Agents)." developers.google.com