I went to find out whether my site needed an llms.txt file.

I came back with two answers and both of them were from Google.

What the file is

An llms.txt is a plain text file at the root of a website, written in Markdown, that describes what the site contains and links to the important parts.

The idea is that an AI agent reading it gets a clean map instead of trying to reconstruct one from HTML full of navigation and advertising.

It is a proposal, not a standard.

Jeremy Howard published it on 3 September 2024 and revised it in August 2026. The current version states what has happened since:

"thousands of sites publish an llms.txt file, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one as part of its agentic browsing checks. The AI labs themselves publish llms.txt files for their own developer docs: OpenAI, Anthropic, and Gemini."

That is a serious claim set. Thousands of sites. Lighthouse auditing for it. And the AI labs using it.

Lighthouse is made by Google. Hold on to that.

So I went to look, and that is where it stopped being simple.

What Google Search says

In its guide on optimizing for generative AI features, Google Search is direct:

"Prioritize effective SEO strategies over 'AEO/GEO hacks': For Google Search, you can ignore tactics like 'chunking' content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions."

Unnecessary is Google's word, about this file, in a guide published in 2026.

That looked like an answer. Then I checked whether Google follows its own advice.

What I actually tested

I requested /llms.txt from 28 large domains and recorded what came back. Two things had to be separated: a site that does not have the file, and a site that blocks the request so you cannot tell either way. A 403 from a firewall is not evidence of absence, so those are marked as undetermined rather than counted against anyone.

HAS ONE                          DOES NOT (clean 404)
  github.com                       google.com
  docs.github.com                  developers.google.com
  stripe.com                       openai.com
  docs.stripe.com                  x.ai
  vercel.com                       bing.com
  cloudflare.com                   nytimes.com
  developers.cloudflare.com        theguardian.com
  shopify.com                      bbc.com
  wordpress.org                    wired.com
  mistral.ai                       theverge.com
  cohere.com                       wikipedia.org
  deepmind.google                  mozilla.org
  developers.openai.com            quora.com
  cookbook.openai.com
  docs.anthropic.com

UNDETERMINED (blocked, 403 even with a browser user-agent)
  perplexity.ai   stackoverflow.com   npmjs.com   medium.com

The first thing this shows is that the proposal is right. OpenAI does publish one, on developers.openai.com. Anthropic does, on docs.anthropic.com. I had checked openai.com first, got a 404, and nearly wrote that the proposal was wrong. It was not. I was looking at the wrong host.

The second thing is the one I did not expect.

Two parts of Google, two answers

developers.google.com/llms.txt returns 404. So does google.com/llms.txt. So does gemini.google.com/llms.txt.

deepmind.google/llms.txt returns a properly written file. It states what the domain is, and links to a second file at /llms-full.txt for what it calls "deep RAG workflows and research context mapping". It is, by the standard the proposal describes, an example of the thing done well.

So:

Google Search       calls llms.txt unnecessary, and does not publish one
Google DeepMind     publishes one, and a fuller one beside it
Chrome Lighthouse   audits your site for one, according to the proposal

All three are Google.

Nothing here is a scandal, and I want to be careful not to make it one. A company that size contains many teams, and teams disagree. DeepMind publishing a directory map for AI agents has nothing to do with whether Search reads the file, and both can be right about their own thing.

But if you are a site owner trying to decide what to do, that is three signals from one company and they do not point the same way.

What actually decides it

There is one question that resolves this, and it is not which team is right.

Who is doing the reading?

An llms.txt is for an agent that fetches a website to answer a question. A coding assistant looking up an API call. A chat tool reading a page about a product.

Those agents read the file if someone hands it to them. That is why the adoption pattern looks the way it does. The sites that have one are almost entirely documentation and developer infrastructure. Stripe, GitHub, Cloudflare, Vercel, Shopify, WordPress. Places where the reader is a machine doing a task.

Publishers do not have them. Not the New York Times. Not the Guardian. Not the BBC.

Their reader is a person, and a person does not read a text file.

And Google Search is not that kind of reader.

Google Search does not need your llms.txt to find or understand your pages. It has its own systems, it says so in plain words, and no file you add changes them.

Which is what "unnecessary" means. Unnecessary for Search.

It is not the same as useless. Those are different claims and the coverage of this file runs them together constantly.

What I did with my own site

I did not add one. Three reasons, in order of how much they mattered.

My reader is a person. This site publishes articles about getting indexed. Nobody arrives with a task that a machine has to complete.

Google Search indexes me, and it does not read the file. The pipeline that decides whether my pages appear is documented, and none of it mentions llms.txt.

The three signals do not resolve into a rule. When one company says unnecessary and publishes one anyway, the honest position is to wait rather than to guess. Adding a file because a proposal says thousands of sites have one is the same reasoning that produced word counts that do not exist, and a definition with no length in it. I have written about that mistake twice already. I am not going to make it a third time on the strength of a file format.

If I ran documentation, I would add one tomorrow. That is the line. Not company size, not whether it is fashionable. Whether the thing reading your site is a person or a machine with a job to do.

The short version

llms.txt is a real proposal with real adoption, and the people behind it are not exaggerating: OpenAI and Anthropic publish one, and thousands of sites do.

Google Search calls it unnecessary, in writing, and does not publish one.

Google DeepMind publishes one, and a fuller version beside it.

Chrome, according to the proposal, audits your site for one.

That is not a contradiction anybody has to resolve. It is a signal about who each part of Google is talking to. Search is talking about indexing, where the file does nothing. DeepMind is talking about agents reading its research, where the file does something.

The useful question is not whether llms.txt works.

It is whether anything reads your site that would use it. For most publishers the answer is no, and no amount of the internet saying otherwise changes it.

Sources

  • The /llms.txt file, v2, Jeremy Howard, published 3 September 2024, modified 10 August 2026
  • Google Search documentation, optimizing for generative AI features
  • Direct requests to /llms.txt on 28 domains, 12 October 2026