What Is robots.txt? How It Controls Search Engine Crawling

robots.txt is a plain text file placed at the root of a domain (e.g., example.com/robots.txt) that tells well-behaved crawlers which parts of a site they are allowed or disallowed to access, using the Robots Exclusion Protocol. Google's documentation is explicit that robots.txt controls crawling, not indexing — a disallowed page can still be indexed and shown in search results (typically without a description) if it's linked to from elsewhere, since Google never crawled it to see a noindex tag in the first place.

Published: Sep 5, 2026 Last updated: Sep 5, 2026 Reviewed by: NOXEL Research Reading time: ~8 min Difficulty: Beginner → Intermediate Primary topic: Technical SEO

Definition

robots.txt implements the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers check before fetching pages from a site. It must be placed at the root of the domain it applies to (a robots.txt at example.com/robots.txt only governs example.com, not subdomains) and be accessible as a plain text file over HTTP or HTTPS. Established

Basic Syntax

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

Each block starts with a User-agent line specifying which crawler the rules apply to (* matches all crawlers that respect the standard), followed by Disallow and/or Allow rules. A Sitemap directive can point crawlers to the XML sitemap regardless of which user-agent block it's placed under. Established

Crawling Control, Not Indexing Control

The most common misunderstanding

Google's documentation states plainly that Disallow in robots.txt prevents crawling, but does not guarantee a URL won't appear in search results. If other pages link to a disallowed URL, Google may still index it — typically showing it with a generic or missing description, since it was never crawled. To reliably keep a page out of search results, Google recommends a noindex meta tag or HTTP header instead, on a page that is crawlable so the tag can actually be read. Established

Not a Security Mechanism

robots.txt is a publicly readable file, and it relies entirely on voluntary compliance — it has no technical enforcement mechanism. Google's guidance explicitly warns against using it to hide sensitive information, since malicious crawlers can ignore it outright, and the disallowed paths themselves are visible to anyone who views the file. Established

robots.txt and AI Crawlers

Beyond traditional search engines, robots.txt has become the primary control point for AI-related crawlers — training crawlers like GPTBot, search-indexing crawlers like OAI-SearchBot and PerplexityBot, and permission tokens like Google-Extended. Each of these is independently controllable via its own User-agent block, meaning a rule for one does not automatically apply to another. Established

Common Mistakes

Assuming Disallow prevents indexing. It only prevents crawling; a linked, disallowed page can still appear in search results.
Placing robots.txt anywhere other than the domain root. A robots.txt file at a subdirectory or subdomain other than the one intended has no effect on the domains it doesn't reside on.
Accidentally disallowing an entire site with Disallow: / under User-agent: *, blocking all compliant crawlers from the whole domain.
Using robots.txt to hide sensitive data. The file itself is publicly readable, and disallowed paths are visible to anyone who checks it.
Blocking one AI crawler and assuming it blocks all AI crawlers. Each user-agent (GPTBot, PerplexityBot, OAI-SearchBot, etc.) must be addressed individually.

FAQ

Does Disallow in robots.txt keep a page out of Google's search results?

Not necessarily. It prevents crawling, but a disallowed page can still be indexed and shown in results if other pages link to it — Google recommends a noindex tag for guaranteed exclusion.

Can robots.txt hide private or sensitive pages?

No. The file is publicly accessible and readable by anyone, and malicious crawlers may ignore its rules entirely. It should not be relied on for security or privacy.

Does blocking one crawler in robots.txt block all crawlers?

No. Rules apply per user-agent. Blocking GPTBot, for example, has no effect on PerplexityBot or Googlebot unless each is addressed in its own block.

Sources

Google Search Central. "Introduction to robots.txt." developers.google.com
Google Search Central. "Robots.txt Rules and Google Search." developers.google.com

← Back to Learn