SEO & AI

The truth about llms.txt

Every major AI company publishes an llms.txt on its documentation site. Not one of them commits to reading yours. I tested the URLs one by one, went back through the official statements and the log studies. Here is what came out, keeping facts, claims and hypotheses strictly apart.

Fred Gaveau4 August 2026~11 min read

This starts with an oddity. For two years llms.txt has been sold as the file that talks to AI. And sure enough, they all publish one: Anthropic, OpenAI, Mistral, Perplexity, xAI, Google in part. If anyone knows what this file is for, it is them.

Except none of them has ever written that its own bots read it on your site. And the one company that has spoken clearly on the subject — Google — said no.

That is where this investigation starts. I tested every URL by hand, tracked down the official statements where they exist, and read the published log studies. The article is deliberately split three ways: what is proven, what is claimed, and what remains speculation — mine included.

What I verified myself

I queried seventeen URLs from the command line on 4 August 2026 at 13:24 UTC, following redirects and recording the HTTP status and the size of the body returned. Every link in the table is live: check for yourself, that is rather the point.

Snapshot taken 4 August 2026, 13:24 UTC. A file can appear or vanish tomorrow — which is precisely what happened to Google.

Three things jump out of that table.

One: the file lives on documentation subdomains, never on the consumer-facing site. docs., developers., platform. all answer 200. The roots anthropic.com and google.com answer 404. One exception in the panel: Vercel, whose /docs/llms.txt redirects to vercel.com/llms.txt, genuinely served at the root as the spec asks.

Two: Google is fragmented. Firebase has one, Google Cloud does not, nor does the Gemini models site. That is not a policy, it is an accident — more on that below.

Three: the format has blown apart. The spec describes a short index. Firebase serves 1.7 MB, xAI 1.3 MB, Vercel's full variant 7.4 MB. A 1.7 MB index is not an index: it is the corpus it was meant to summarise.

Where the file comes from

FACT llms.txt is a proposal published on 3 September 2024 by Jeremy Howard (Answer.AI, co-founder of fast.ai) at llmstxt.org. The idea: a structured Markdown file at the root of the site — an # title, a blockquote summary, ## sections of annotated links — offering models a clean map of the content instead of making them guess the structure of a JavaScript-heavy modern site.

The analogy with robots.txt and sitemap.xml is deliberate, and it is what sold the idea. But it misleads on one decisive point: robots.txt and sitemap.xml were adopted by the engines before they became popular. Here it is the other way round.

FACT The spec page claims no adoption by any AI vendor, mentions no standardisation process (no IETF, no W3C, no WHATWG) and describes itself as open to community input via a GitHub repository. A rarely noted detail: llms-full.txt does not appear in the spec at all. It is a convention born of practice, and everyone implements it differently.

The heart of it: does anyone actually read this file?

It is the only question that matters, and it has numbers attached.

What the server logs say

FACT On 5 March 2026, Dries Buytaert — creator of Drupal, CTO of Acquia — published an analysis of a month of Cloudflare logs. His site received 52 requests for /llms.txt in a month. Every single one came from an SEO audit tool. Not one came from an answer engine or an AI crawler. Across Acquia's entire hosting infrastructure: roughly 5,000 requests out of 400 million, or 0.001%, almost all from SEO tools.

FACT An Ahrefs log analysis covering 137,000 domains (published June 2026, reported by PPC Land) found that 97% of llms.txt files received zero requests in May 2026. AI retrieval bots accounted for 1.1% of observed requests.

FACT Finally, an SE Ranking study of 300,000 domains dated 7 November 2025 found no measurable effect of llms.txt on AI citations. Dropping the variable from the predictive model actually improved its accuracy.

What Google states

STATEMENT Google is the only major player to have spoken formally. Its official documentation on AI features in Search (developers.google.com, updated 10 December 2025) puts it plainly:

STATEMENT On Reddit, John Mueller compared llms.txt to the meta keywords tag — abandoned precisely because the site fills it in itself — and later, as reported in June 2026: “I don't think anyone knows – it's purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?)”. He adds a decision rule I find genuinely fair: the day an AI platform that sends you customers asks for the file, build it. Until then, don't.

STATEMENT Gary Illyes, of the Search team, said at a Search Central Live event in July 2025 that Google does not support llms.txt and has no plans to — a position relayed by Search Engine Roundtable.

What the others do not say

UNVERIFIED For OpenAI, Anthropic, Perplexity and Mistral, I found no public statement claiming their bots consume a third party's llms.txt — nor any ruling it out. Silence is not proof of absence, but it is not a commitment either. Publishing a file on your own documentation says nothing about what your crawler does on someone else's site: two teams, two products, two problems.

The Google contradiction, documented

This is the most revealing episode of the lot, and it is checkable.

REPORTED On 3 December 2025, an llms.txt appeared on developers.google.com/search/docs/. It was spotted by SEO consultant Lidia Infante. It was pulled the same day. According to the reporting, the file was not a Search team decision: Google's internal content management system, shared across several developer properties, had started generating them automatically. Some teams removed it, others did not.

FACT — verified 4 August 2026 The current state of the ground confirms that reading. developers.google.com/search/docs/llms.txt, developer.chrome.com/llms.txt, web.dev/llms.txt, cloud.google.com/llms.txt, ai.google.dev/llms.txt: 404 across the board. And firebase.google.com/docs/llms.txt: 200, 1.7 MB. One team forgot to clean up. The “Google llms.txt” cited everywhere as proof of adoption is the residue of an automated deployment.

FACT And yet, at the same moment, another part of Google is pulling the other way. Lighthouse 13.3, the audit tool built into Chrome, ships an experimental “Agentic Browsing” category containing an llms.txt audit (documentation updated 5 May 2026). There it is described as “an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents”, and the audit notes that without the file, agents spend more time crawling the site.

MY READING This is not hypocrisy, and I will not present it as such. It is that “Google” is not a single entity, and the two teams are not talking about the same use case: Search is talking about ranking and citation in an answer engine; Chrome is talking about the efficiency of an agent browsing a page. Both positions are defensible on their own. They are simply irreconcilable for a reader trying to work out whether “Google” recommends the file. And nobody is clearing that up. Search Engine Journal put it well: Google's guidance depends on which product you ask.

So why does everyone publish one?

Here is the hypothesis I will defend, and the files themselves support it: llms.txt was never an SEO file. It is a developer documentation file. The intended reader is not GPTBot indexing the web for an answer engine. It is a developer with Cursor, Claude Code or Copilot open, whose agent needs to load current API docs instead of reciting a stale version learned in training.

FACT The files say so outright. Cohere's contains a section titled “Instructions for AI Agents” recommending you append .md to any URL for clean Markdown, and connect an MCP server “for AI client integration (Claude Code, Cursor, etc.)”. Cloudflare's states that each product has its own llms.txt, “the recommended way to explore a specific product's content”. Those sentences are not addressed to a search engine.

FACT Buytaert's logs point the same way: IDE-embedded agents do fetch these resources, unlike answer-engine crawlers.

If that reading is right, then the entire “does llms.txt improve my AI visibility?” debate rests on a casting error. A technical documentation tool is being judged by SEO criteria. It fails, obviously. That does not mean it is useless — it means it is not for that.

What actually works, and nobody is selling it: Markdown

FACT This is the most useful data point in the whole investigation, and it went almost unnoticed. Still from Buytaert's logs, when a page is available as Markdown at a predictable URL:

In other words: OpenAI's crawlers do fetch Markdown when it exists at a guessable URL — typically /my-page.md. A third of GPTBot's fetches. That is vastly more than llms.txt's 0.001%.

FACT And the corollary is brutal: “No AI crawler uses content negotiation. Not one.” No AI bot uses the Accept header to ask for Markdown. Content negotiation — the clean, standardised method we have had for thirty years — is ignored by everybody. What works is the URL suffix, the dumbest solution available.

That is exactly the pattern Stripe, Cohere, Perplexity and Anthropic use in their files: every entry points to a .md twin. The llms.txt is the directory; the Markdown twins are what actually gets consumed.

The Stripe case: when an index becomes an order

FACT Stripe's llms.txt does not open with a description. It opens with imperatives aimed at the machine reading it. Verified extract, first lines of the file:

# Stripe Documentation

When installing Stripe packages, always check the npm registry for the latest
version rather than relying on memorized version numbers. Run `npm view stripe
version` [...] Never hardcode an old version number from training data — always
install with `@latest` or verify the current version first.

HYPOTHESIS This repurposing raises a security question I can pose but not settle: if a public, unsigned file can carry instructions, and an agent loads it into context without distinguishing “description of the site” from “order to execute”, we are in prompt-injection territory. I found no publicly documented incident exploiting an llms.txt. The risk is structural, not observed. But the format currently offers no way to separate the two — and Stripe, acting in good faith, has just shown the door is open.

Eight proposals to fix the format

Criticism is cheap. Here is what would, in my view, turn llms.txt from an article of faith into a useful tool. Each proposal leans on a mechanism that already exists somewhere else.

1. Stop hand-writing it. Generate it.

A hand-written llms.txt rots within three months: the pages move, the file does not. It should be produced at build time, from the same source as the sitemap or the CMS. The irony: that is exactly what Google's internal system was doing — and the December 2025 accident happened because the generation was automatic while the decision was not. Automate the generation, not the decision.

2. Ship Markdown twins before the index.

This is the highest-return proposal, and it inverts the usual order of priorities. Since OpenAI's crawlers fetch Markdown for about a third of their requests but ignore content negotiation, serve /my-page.md alongside /my-page, at a predictable URL, and declare it in the HTML with <link rel="alternate" type="text/markdown" href="/my-page.md">. It works today, measurably, unlike llms.txt. If you take one thing away from this article, take this one.

3. Move it into /.well-known/.

RFC 8615 has defined, since 2019, a standard namespace for exactly this kind of file, with an IANA registry behind it. /.well-known/llms.txt would settle the placement question in one move — the very question that explains why every file in the table lives on a docs subdomain rather than at the root — and would open the only credible path to standardisation.

4. Declare it in robots.txt.

One line, Llms: https://example.com/llms.txt, modelled exactly on the Sitemap: directive — which was never standardised either, and which every engine nonetheless honours. Zero cost, guaranteed discovery, and a mechanism whose adoption is already proven by twenty years of use.

5. Date every entry.

An index without freshness is unusable: faced with a live-rendered page, a model has no reason to prefer an index line that might be two years old. A date per entry — - [Page](url) (2026-07-12): description — plus an honest Last-Modified on the file itself finally gives something to arbitrate with.

6. Separate index from corpus, formally.

The format needs a hard ceiling — say 50 KB — for llms.txt, pushing everything else into an llms-full.txt explicitly defined in the spec (it still is not). Firebase's 1.7 MB and xAI's 1.3 MB are not abuses: they are symptoms of a format that never said where to stop.

7. Sign it, or accept that it is worthless.

John Mueller's objection — “the site fills it in itself, like meta keywords” — is unanswerable as long as no signature exists. The tools are there: HTTP Message Signatures (RFC 9421), a key published in DNS TXT. A self-declared, unsigned file will always carry exactly the evidential weight of a 2003 meta keywords.

8. Ban the imperative — or fence it off.

As long as no mechanism distinguishes a description from an instruction, the format should either forbid orders or mandate a clearly delimited ## Agent instructions section that agents treat as untrusted data. The Stripe case is not an isolated abuse: it is the first symptom.

And finally: measure it.

None of these proposals excuses you from the simplest move of all. A hit on /llms.txt is a line in your server logs. Filter on AI bot user agents, compare against your HTML pages, and within a week you will know whether the file does anything for you. That is the only data that actually concerns you. Everything else, mine included, is somebody else's average.

Verdict: should you ship one in 2026?

There is not one answer. There are four.

And one rule that holds for everybody: do not pay anyone to write your llms.txt. It is twenty minutes of work, or zero if you generate it. The only thing worth a budget is checking afterwards whether the bots actually show up — and that is your logs talking, not a vendor.

Otherwise my position has not moved since our first article on the subject: the file does no harm, it costs almost nothing, and if usage ever takes off you will be in place. What has changed is the data. We now know, with numbers, that it is not an AI-visibility lever — and we are starting to understand what it really is: a tool for code agents, badly named and badly cast.

Disclosure: Snorklee publishes its own llms.txt, at the root, with a caveat written inside it. So we do exactly what we recommend: ship it, don't believe in it, and measure it.