What Is llms.txt?
llms.txt is a proposed plain-text/markdown file, placed at a site's root, that offers AI systems a short, curated summary of a site plus links to its most important pages. It was proposed by Jeremy Howard (Answer.AI) in September 2024. As of mid-2026, no major AI provider — OpenAI, Anthropic, or Google — has made a formal commitment to parse it, and server-log research shows the major AI assistants it was designed for are barely reading the files that exist. Established
How It's Different From robots.txt and a Sitemap
The three files answer different questions and none replaces the others:
| File | Question it answers | Format |
|---|---|---|
| robots.txt | What is this crawler allowed to access? | Directive rules |
| sitemap.xml | What are every one of my indexable URLs? | Comprehensive XML list |
| llms.txt | What should an AI system read first to understand this site? | Curated markdown summary |
A typical llms.txt starts with an H1 site name, a one-sentence blockquote summary, then labeled link-list sections (about, docs, policies) pointing to a deliberately small subset of pages — not every page on the site.
Real-World Adoption vs. Real-World Use
Adoption of the file itself has grown quickly: independent tracking by Originality.ai found roughly 4,000 llms.txt files in mid-2025, growing to over 36,000 by May 2026 — an 8.8x increase in a year. Established
A separate June 2026 study by Ahrefs, based on server logs from 137,000 domains, found that 97% of llms.txt files received zero requests in the month studied. Even where AI retrieval bots were active, they accounted for just 1.1% of the requests to these files — the top requesters were SEO audit tools and general web crawlers, not the consumer AI assistants the format targets. GPTBot accounted for 4.51% of requests and ClaudeBot for 0.80%. Strong evidence (single large study)
One notable, more recent signal: Google added llms.txt to Chrome Lighthouse's new "Agentic Browsing" audit category on May 5, 2026. This shows Google's tooling is starting to check for the file's presence — it is not confirmation that Google's own AI systems consume its contents for retrieval. Established
Common Mistakes
- Treating llms.txt as a ranking or citation lever. There is currently no evidence major AI answer engines weight it in retrieval or citation decisions.
- Assuming it replaces robots.txt. It has no access-control function — it can't block or allow a crawler.
- Assuming it replaces the sitemap. It's intentionally a curated subset, not a comprehensive URL list — search engines still need your sitemap.
- Spending significant time perfecting it. Given current read rates, it's reasonable to publish a simple, accurate version and move budget to things crawlers demonstrably do use — like fixing robots.txt/firewall access issues or structured data.
NOXEL's Own Approach
NOXEL SEO publishes an llms.txt file on noxelseo.com. It costs little to maintain and does no harm, consistent with the current evidence: worth having, not worth treating as a visibility strategy on its own.
FAQ
Do OpenAI and Google actually read llms.txt?
As of mid-2026, neither has made a formal commitment to parse it. June 2026 server-log research found AI retrieval bots made up only 1.1% of requests to these files, and 97% received zero requests at all.
Is llms.txt a replacement for robots.txt or my sitemap?
No. robots.txt controls access, a sitemap lists every indexable URL, and llms.txt is an editorial summary of a small subset of pages. None substitutes for the others.
Should I still publish an llms.txt file?
It's low-cost and not harmful, but current evidence shows major AI crawlers largely aren't reading it yet. Treat it as optional and low-priority, not a visibility strategy.
Is anyone actually using llms.txt data right now?
Some smaller AI-focused tools and developer tooling reportedly consume it. 2026 data found the top requesters were SEO audit tools and general web crawlers, not major consumer AI assistants.
Related Concepts
Part of The Complete Guide to AI Search Visibility (2026)
Sources
- llmstxt.org. "The /llms.txt file." llmstxt.org
- PPC Land. "llms.txt adoption rises 8.8x but 97% of files get zero AI requests" (June 2026, citing Originality.ai and Ahrefs data). ppc.land
- PPC Land. "Google adds llms.txt to Chrome Lighthouse as agentic web standards heat up" (May 2026). feedbagel.com