AI Crawlers Explained: GPTBot, OAI-SearchBot, PerplexityBot and Others
Each major AI platform uses one or more dedicated crawlers, independently controllable in robots.txt. OpenAI uses three (training, search indexing, and live user-triggered fetching). Perplexity uses two (indexing and live fetching). Google's model differs: Google-Extended is a permission token, not a real crawler, and Google's AI Overviews run on the regular Googlebot with no separate robots.txt control. Blocking the wrong one — or assuming they're all the same — is one of the most common causes of unexpected AI invisibility.
Crawler Reference Table
| Platform | Crawler | Purpose | Blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | Opts out of training only |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT Search | Removes eligibility for ChatGPT Search citations |
| ChatGPT-User | OpenAI | Live fetch when a user asks ChatGPT to read a URL | Blocks that specific live-read feature |
| PerplexityBot | Perplexity | Builds the index behind cited answers | Removes eligibility for Perplexity citations |
| Perplexity-User | Perplexity | Live fetch on user request | Blocks that specific live-read feature |
| Google-Extended | Permission token for Gemini/Vertex AI training | Opts out of training only — not a real crawler | |
| (regular) Googlebot | Powers Search, and also AI Overviews / AI Mode | Blocking it removes a site from Search entirely, AI Overviews included |
OpenAI's Three Crawlers
OpenAI's own developer documentation and Help Center confirm three separate, independently controllable user agents: GPTBot, OAI-SearchBot, and ChatGPT-User. A publisher can allow OAI-SearchBot for search visibility while disallowing GPTBot to keep content out of model training — the settings don't affect each other. Established
OpenAI's documentation notes a nuance worth knowing: even if a page is disallowed, OpenAI states it may still surface the page's link and title in ChatGPT Atlas if the URL is obtained from a third-party search provider or by crawling other pages — full exclusion requires a noindex meta tag, which itself requires the page to be crawlable in the first place for the tag to be read. Established
Perplexity's Crawlers
Perplexity documents two user agents: PerplexityBot for building its search index, and a separate agent for live, user-triggered fetches. Perplexity's published documentation states that PerplexityBot respects robots.txt directives. Established
In June 2024, an independent investigation by developer Robb Knight found server log evidence in a specific case where Perplexity appeared to access content disallowed by robots.txt without presenting its declared PerplexityBot user-agent string. This is a documented incident from one investigation, not an official acknowledgment from Perplexity or proof of an ongoing, universal pattern — but it's worth knowing that declared crawler behavior and observed behavior aren't always guaranteed to match for every AI platform. Strong evidence (single documented incident)
Google's Different Model
Google-Extended is not a fetching crawler — it's a permission token layered onto Google's existing crawling infrastructure. Content is still fetched by regular Googlebot; Google-Extended only governs whether that already-crawled content may be used to train Gemini and Vertex AI models. Established
Google's AI Overviews and AI Mode run on the same regular Googlebot crawl and index used for standard Search results — there is no separate robots.txt token for them. If content is accessible to Google Search, it is currently also accessible to AI Overviews; blocking Google-Extended has no effect on this. Opting out of AI Overviews specifically requires a different control inside Google Search Console, not a robots.txt directive. Established
Example robots.txt Configuration
A configuration that allows AI search citations while opting out of model training, across OpenAI and Perplexity:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Disallow: /This blocks training-oriented crawlers while keeping search-indexing and live-fetch crawlers open. Adjust based on your own position on AI training versus AI search visibility — there is no universally "correct" configuration.
Common Mistakes
- Blocking GPTBot and assuming it also blocks OAI-SearchBot. They're independent — this removes training exposure but leaves ChatGPT Search access untouched, or vice versa depending on which one is actually blocked.
- Blocking Google-Extended expecting it to affect AI Overviews. It doesn't — AI Overviews run on regular Googlebot.
- Assuming every declared crawler behavior is independently verified. Treat robots.txt compliance claims as the platform's stated policy, not a guarantee, and monitor server logs if precise control matters.
FAQ
Does blocking GPTBot remove me from ChatGPT Search?
No. GPTBot and OAI-SearchBot are independent crawlers controlled separately in robots.txt.
Does blocking Google-Extended remove my content from AI Overviews?
No. Google-Extended only controls Gemini/Vertex AI training. AI Overviews run on regular Googlebot, controlled via Search Console rather than robots.txt.
Do AI crawlers always respect robots.txt?
Most documented AI crawlers state that they respect robots.txt. At least one independent investigation found log evidence suggesting an exception in a specific case — worth knowing, not proof of a universal pattern.
Related Concepts
Sources
- OpenAI Help Center. "Publishers and Developers – FAQ." help.openai.com
- OpenAI Developer Docs. "Overview of OpenAI Crawlers." developers.openai.com
- Perplexity. "Bots / Perplexity Crawlers." docs.perplexity.ai
- Google Search Central. "Google Crawlers (User Agents)." developers.google.com
- Knight, R. (2024). "Perplexity AI Is Lying about Their User Agent." rknight.me — independent investigation, not a platform-official source.