llms.txt: does anyone read it?

What llms.txt is

llms.txt is a markdown file at your site root that lists the pages you would like an AI agent to read, with short descriptions. Jeremy Howard proposed it in September 2024 and a second version was published on 10 August 2026. The proposal is explicit about its own status.

A proposal, open for comment.

When you are sold this file, it usually arrives under another name. The work gets called answer engine optimization, and llms.txt is the deliverable most often attached to it.

Its stated purpose is to standardise on using an llms.txt file to provide information to help agents use a website. The format is an H1 heading, an optional summary in a blockquote, and H2-delimited lists of links. It sits at the root, or at any path, covering the pages under that path. It sits alongside the rest of the work in AI search engine optimization and replaces none of it.

The proposal notes that the AI labs publish llms.txt files for their own developer documentation, naming OpenAI, Anthropic and Gemini. That is a fact about their docs sites, not a commitment from their crawlers. The difference between those two things is where the rest of this goes.

Seven measurements of two different questions

How widely is it used splits into two questions that get quoted as one: how many sites publish the file, and whether anything fetches it. Several organisations have counted, on both, and the answers range across an order of magnitude.

Each of them counted a different web.

Web Almanac, Presenc, Ahrefs, NerdyData, Semrush, OtterlyWhy two true adoption numbers can be twenty times apart

Seven published figures, read on 2 September 2026. Each row carries the universe it counted, because that is the whole explanation.

Why the spread is not a scandal

A crawl of the whole web and a sample of domains that received traffic are not the same population. Ahrefs measured domains in its own analytics that received traffic in May 2026; the Web Almanac crawls far more of the web, including sites with no traffic at all. Why the shares differ in that direction is a reasonable inference. Neither study tested it. Ahrefs measured 137,210 domains that received traffic in May 2026 and found 28 percent publishing one. The Web Almanac, crawling far more of the web, lands near 2 percent. Both can be right, and neither answers whether the file does anything.

97 percent received zero traffic

In the same study, Ahrefs reported that 97 percent of those files received zero traffic in May. Semrush ran a narrower and more direct test on Search Engine Land: between mid-August and late October 2025 the file received zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot. Semrush also reports Google’s John Mueller saying flatly that no AI system currently uses llms.txt.

Otterly’s experiment is the one result pointing the other way, with 62,100 AI-bot visits observed over 90 days. That is a different setup and a different measurement, so the matter is not settled in either direction.

Three mistakes in llms.txt advice

Two measurements pointing different ways is a reason to look at what each one counted, not a reason to pick a side. The pattern under most llms.txt advice is a category error: treating a file that describes your site as a file that instructs a crawler. A robots.txt file instructs. This one does not.

Reading a tool check as a requirement

Lighthouse added an Agentic Browsing category with an llms.txt audit, and that fact travels as evidence the file is expected. Read the audit itself and it says something narrower.

Treating publication by an AI lab as consumption by its crawler

OpenAI and Anthropic publishing llms.txt for their own developer documentation tells you they think it is useful for agents reading docs. It says nothing about whether their crawlers fetch yours, and their own crawler documentation does not mention the file. OpenAI’s publisher guidance is to allow OAI-SearchBot in robots.txt; Perplexity’s is to use robots.txt tags. Neither names llms.txt.

What one looks like when a large site ships it

1. 1The file is markdown links, not markup and not a feed. Every line is a product name, a URL and one sentence.
2. 2The pattern here is nesting: the top file points at a per-product llms.txt, not at the pages themselves.
3. 3This is a documentation site, which is the case the format was designed for. Nothing on it resembles a product catalogue or a category page.

Spending on it instead of on access

The file cannot help a crawler that is blocked, and blocking is the one failure here whose mechanism every platform documents. Whether your robots.txt lets the named agents through is a question for an ecommerce SEO audit, and it is the check that comes first. Separately, Core Web Vitals carry the only published pass mark in this area, and Google does not name them as an AI-feature eligibility condition.

What to do about it on a store

It costs almost nothing and nobody has measured its effect. Treat it as a small bet, not a strategy. Robots directives, which do have documented consequences, come first.

1. Check the crawlers can reach you at all

   Open robots.txt and look for OAI-SearchBot, PerplexityBot and the other named agents. A block there is a documented reason a crawler would not fetch you, though Google notes a blocked page can still appear in results when others link to it. A missing llms.txt is not a documented reason for anything.

2. Publish one if it is generated, and do not hand-maintain it

   Documentation platforms generate the file from content that already exists. On a storefront, a hand-written list of pages goes stale the week the catalogue changes, and nobody has measured what a stale one costs. What is certain is the maintenance: a recurring task for a file nothing has been shown to read. The same argument applies to the product feed behind Google Shopping ads, which the platform has been generating for years.

3. Keep it small and point at pages that answer questions

   The proposal is a reading list. On a store that means guides and category explanations, not every product URL. The same judgement about which pages deserve to exist runs through ecommerce SEO.

4. Do not remove anything else to make room for it

   Google states that eligibility for its AI surfaces is being indexed and eligible for a snippet, and AI Overviews sets out what that means. Nothing about llms.txt changes that requirement.

5. Set a review date, not a monitoring habit

   The published measurements will change. Reading them again in six months costs less than watching your own log, where a minority of files see any traffic at all and the traffic that does arrive cannot tell you what it changed.

Why the recommendation is this small

Two things would change it. The first is a platform saying it reads the file: OpenAI and Perplexity both publish crawler documentation and neither mentions it, so that statement does not exist yet. The second is a published measurement connecting a fetch to a citation or a visit, and none of the seven measurements above attempts that link.

Until one of those arrives, the file is a cheap option on a future that may not come. Buying a cheap option is reasonable. Budgeting for it as a channel is not, and the difference is the entire practical content of this page.

That is a deliberately small recommendation. The recommendation is small because the measurements are. Most of these files see no traffic in a month, a minority do, and Otterly reports tens of thousands of bot visits in its own setup. A thing fetched sometimes, by unclear agents, with no published link to an outcome, is not a lever yet.

What llms.txt costs

Generating the file is close to free on a platform that does it automatically, and a small manual task otherwise. The cost that matters is not the file. It is what gets displaced when a proposal is sold as a requirement.

The file is cheap. The framing is not.

Four claims are each true in one reading and misleading in another. The question that separates the readings is short in every case:

• Adoption is at 28 percent. Of which population, measured how.
• The AI labs use llms.txt. For their own documentation, or when crawling mine.
• Lighthouse checks it. What happens to the score when the file is missing.
• It improves AI visibility. Which published measurement links a fetch to a citation.

The last one has no published answer yet, which is where the subject currently stands.

What we would ask a supplier proposing it: which measurement are you relying on, and does it measure adoption or consumption. The two are usually quoted in the same sentence and they answer different questions. The label the work is sold under, answer engine optimization, is an industry term and not a platform one. Whether to fund it before ordinary search work is settled in AEO vs SEO.

Say you want your pages readable to an assistant with or without the file: that is our LLM optimization service.

How you would know if it started working

The measurement is easy and almost nobody does it. The answer sits in your own server logs, not in anyone else’s study. Three lines, and the third is where the subject runs out.

1. Requests to /llms.txt in your server logs, by user agent. That is the question, and on a server whose logs you can read it is a filter, not a tool. On hosting that does not expose raw logs, it needs a CDN log export or nothing at all.
2. Whether those requests come from named AI agents or from general crawlers and scrapers. Ahrefs found that of the fetches that did happen, retrieval bots were a small share of AI bot requests.
3. Whether anything downstream changes. A file being fetched is not a citation, and no published measurement yet connects the two.

The third line is where the subject runs out. Nobody has published a measurement linking an llms.txt file to a citation or to a visit, in either direction.

There is one number about your own site in all of this. Grep your access logs for llms.txt and count the requests in the last thirty days. Whatever number comes back is worth more than every adoption statistic in this field, because it is the only one measured on your site.

The citation measurements that do exist were made without this file in the picture, and they are set out in AI search engine optimization.