The llms.txt debate: signal or theatre?
10% adoption across 300,000 domains, but AI search crawlers almost never fetch the file. Google has explicitly said it ignores it. Yet every serious developer-tool site is shipping one anyway. We read the data.
Not an AI search play in 2026. A legitimate developer-tooling play. Low cost, non-zero upside. Ship it — but don't count it as a GEO tactic.
Llms.txt is a proposed standard: a plain-text Markdown file at the root of a domain that provides AI systems with a curated list of the site's most important content. Proposed in September 2024 by Jeremy Howard, it was positioned as the AI-era equivalent of robots.txt — a way to tell language models which pages matter most. By mid-2026, it has accumulated both a passionate fanbase and a skeptical counter-argument grounded in data. We reviewed every major study to give you a clear picture.
The adoption data
A SE Ranking study of 300,000 domains found a 10.13% adoption rate — roughly one in ten sites. After 18 months of industry conversation, adoption has remained roughly flat. Notably, adoption was nearly identical across low-, mid- and high-traffic site tiers, which suggests the file is deployed by technically-inclined teams regardless of site scale.
By sector, early adopters were developer-facing SaaS and cybersecurity companies. Financial services and healthcare adoption remains below 10% of top-100 domains, primarily due to legal and compliance conservatism about publishing any machine-readable content policy without formal standardization.
What AI crawlers actually do with it
Limy monitored over 500 million LLM bot traffic events across the brands they track. The finding is blunt: AI search crawlers almost never fetch /llms.txt. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and crawl HTML directly. Of 500M AI crawler visits tracked over 90 days, only 408 targeted llms.txt directly.
Google's position is the clearest. Gary Illyes confirmed in July 2025 that Google doesn't support llms.txt and isn't planning to. John Mueller compared it to the discredited keywords meta tag. No major LLM provider — OpenAI, Anthropic, Google, Meta, Mistral — has publicly committed to using llms.txt as a signal in production search or answer surfaces.
"8 out of 9 sites saw no measurable change in traffic after llms.txt implementation."
Search Engine Land analysis, 2026
The ALLMO analysis of 94,000+ cited URLs found no measurable citation uplift associated with llms.txt adoption. An XGBoost model trained on AI-citation data improved when the llms.txt variable was removed — meaning the file added noise rather than predictive signal.
The verdict: not an SEO play, but not nothing
The practitioner consensus that's emerging is more nuanced than the initial hype. llms.txt is not (yet) an AI search signal. It is, however, a legitimate developer-tooling play. IDE agents — Cursor, Windsurf, Claude Code, GitHub Copilot — do fetch llms.txt when pointed at documentation sites. The standard is increasingly the interface through which code assistants access library documentation.