What is llms.txt? (And why Google says you don't need one)
llms.txt is a Markdown file at your site's root that's supposed to hand your pages to AI models. Ahrefs checked 137,210 domains: 97% of these files are never requested, and most bots that do fetch one aren't AI tools. Here's what it is, who reads it, and what Google says.
llms.txt is a plain text file in Markdown format that sits in your site's root directory — yoursite.com/llms.txt — and lists your most important content so large language models can read it without parsing your HTML. That's the proposed standard, put forward in September 2024 by Jeremy Howard of Answer.AI as a standardized method for handing your website information to a machine. Within a year it had a spec, a validator ecosystem, generators, WordPress plugins and a permanent slot on every AI SEO checklist. It is also, as far as anyone can measure, almost entirely unread.
- A proposed standard, not a supported one. A Markdown file at your root, listing your key pages for large language models.
- Almost nobody reads it. Ahrefs checked 137,210 domains: 97% of llms.txt files got zero requests, and 77% of the bots that did fetch one weren't AI tools at all.
- Google says you don't need it. Its AI-optimization documentation states that Google Search doesn't use machine-readable files, AI text files or Markdown of this kind.
- robots.txt is the file that works. It tells crawlers what they may not fetch, and every major AI crawler honours it. llms.txt carries no permissions at all.
- One honest exception: developer docs read by coding agents. That's a real mechanism. Game discovery isn't.
If you're an indie dev with a game site and a to-do list you'll never finish, this is one item you can cross off. Here's the whole picture, so you can cross it off for the right reasons.
The llms.txt file: basic structure
The format is simple, which is part of why it spread so fast. A single file, no build step, no tooling — you can write it in any text editor, Notepad included.
What goes in the markdown file
The precise format is three things: an H1 with your site name, a blockquote with a short description of what the site is, and sections of links, each with a one-line note on what the page contains — the key information, nothing else.
That's it. The whole spec fits on a napkin, and that's a feature. It's written for machines, not human readers: no styling, no navigation, no images. LLM-friendly plain text, and nothing more.
Where the txt file goes: your root directory
At the root, and nowhere else: yoursite.com/llms.txt. Same convention as robots.txt and sitemap.xml — a small file at a predictable, accessible location, so most crawlers can find it without being told where to look. Put it in a subfolder and it may as well not exist.
llms-full.txt: two files, not one
There's a companion. Instead of a list of links, llms-full.txt holds the entire content of your own website as one long Markdown document — detailed information rather than a table of contents. Two files, same idea, different resolution.
llms-full.txt is far too big for most context windows. A model that can't fit your file doesn't read half of it — it reads none of it.Why the proposed standard exists: more context, less noise
The underlying problem is real. Web content is noisy — navigation, cookie banners, script tags, markup that means nothing to a model. Context windows are finite. Converting complex HTML pages into LLM-friendly content genuinely does produce something easier for AI models to consume, and quick access to a curated index beats crawling forty pages.
The pitch is that this providing-context step reduces inaccuracies in AI generated responses: point the model at trusted, current sources and it stops guessing from a rendered page when it tries to answer questions about you.
Nice idea. A standardized method only works if the other side implements it.
Does Google support llms.txt?
No, and it has said so repeatedly, in public, without hedging.
"FWIW no AI system currently uses llms.txt." He compared it to the meta keywords tag: the tag site owners dutifully filled in for years while search engines quietly ignored it.
Google does not support llms.txt and has no plans to — as reported by Search Engine Land, alongside the reminder that ordinary SEO is what gets you into AI Overviews.
"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."
That last one is the one to bookmark. Not a tweet, not a conference quote out of context — Google's published guidance, saying the quiet part in plain words.
Who actually fetches llms.txt?
This is the part of the Ahrefs study that deserves more attention than the headline number.
The file is gaining traction — that part is true. In July 2025 only 951 domains had published one; by May 2026 the count was over 36,000, an 8.8× rise in twelve months. Adoption is not the problem.
Of the roughly 38,000 domains Ahrefs found with a valid file, about 1,100 saw any requests at all.
Which AI crawlers request the file
When Ahrefs looked at who was doing the requesting, 96% of requests came from bots — and 77% of those bots weren't AI tools at all:
So the honest summary isn't "AI ignores your llms.txt". It's worse, and funnier: the main consumers of llms.txt are the tools that check whether you have an llms.txt. An entire ecosystem of validators, scorers and generators grew up around a proposed standard before anyone established that the standard is being read.
Check it on your own site: server logs
You don't have to take Ahrefs' word for it. Grep your server logs for /llms.txt and look at the user agents that show up. That's the whole experiment, it takes a minute, and it costs nothing. Whatever fetched your file is in there — and on most sites, so is the answer that nothing did.
llms.txt vs robots.txt: different jobs
These get conflated constantly, so let's be precise — because one of them absolutely does work.
A thirty-year-old convention, honoured by every major crawler including GPTBot, ClaudeBot, PerplexityBot and Google-Extended. The only file on this page with enforcement behind it.
Carries no permissions, grants no rights and withholds none. It cannot allow or block anything. A recommendation — and recommendations only work if someone is listening.
| robots.txt | llms.txt | |
|---|---|---|
| What it does | Blocks or allows crawling | Suggests reading priority |
| Who honours it | Every major crawler, AI included | Effectively nobody, per the fetch data |
| Enforcement | Convention, universally respected | None |
| Worth your time? | Yes | Not for discovery |
If your actual goal is controlling what AI crawlers do with your game's website — blocking them, or deliberately allowing them — robots.txt is the file that does that. Google-Extended is the directive that governs Google's AI training use specifically. If you have proprietary content you'd rather AI companies didn't train on, that's the lever. llms.txt is a suggestion box nobody empties.
Do I need llms.txt?
No. Not for Google, not for AI-driven search, not for getting your game mentioned by ChatGPT. For most websites the file is a new file to maintain and nothing else.
It's a list of links. It grants nothing and forbids nothing. The file that controls AI crawlers is robots.txt, and it has worked for thirty years.
Publication grew 8.8× in a year. Requests didn't follow: 97% of files get zero. A standard is only real when the other side implements it, and no major AI platform has.
True, and worth naming. If your users paste your docs into an assistant, or point a development environment at them, a clean Markdown version of your programming documentation genuinely helps — the agent gets LLM-readable text and code blocks instead of a rendered docs site. That's a real mechanism: someone deliberately fetching a file they were told about. It is not discovery.
If you want the file anyway — ten minutes, doesn't hurt, and standards do sometimes get adopted late — then publish it. Just don't let it displace work that has a mechanism behind it.
How to create llms.txt (if you still want one)
No generator needed. Open a text editor, save the file, upload it to the correct directory — the root of your own site, not a subfolder:
# Studio Name
> One sentence on the studio and the game.
## Game
- [Game name](https://yoursite.com/game/): genre, one line on what makes it different.
- [Devlog](https://yoursite.com/devlog/): development updates since 2024.
## Studio
- [About](https://yoursite.com/about/): who we are.
- [Press kit](https://yoursite.com/press/): assets, fact sheet, contact.
That's the entire format. Two important notes: it belongs at the root, and no txt generator is going to do this better than you will — the ones ranking for that term wrap ten minutes of typing in a signup form. If you ship it, keep it current. A file that lists pages you deleted is worse than no file.
What actually moves the needle
If the goal is an AI model describes my game correctly when someone asks, the mechanism runs through what's on your pages and what other people say about you — not through a file in your root directory.
What AI models can actually read on your page
These aren't steps in an order. They're four independent things, and any one of them beats a file nobody fetches.
- Server-render anything that carries meaning. GPTBot and ClaudeBot fetch JavaScript but never execute it. If your feature list is client-rendered, it doesn't exist for them. The ninety-second test is in Humans hate text. AI needs it.
- Get named in places that get read. Mentions, not markup. Coverage, forums, wikis, roundups — that's where artificial intelligence learns your game exists.
- Keep robots.txt sane. Check you haven't accidentally blocked the crawlers you want.
- Write the answer in text. If a fact only lives in a screenshot or a schema block, no model has it.
Generative engine optimization, stripped of its acronym, is mostly this: make sure the information is on the page, in text, and correct — so that when user queries land on your game, the model has something true to say. Keep it in proportion, though. AI referrals are a fraction of a percent of most sites' traffic, and no storefront counts web traffic as a ranking signal anyway. The full argument, with the numbers, is in AI search visibility won't sell your indie game.
Common questions
What is llms.txt in simple terms?
Does Google use llms.txt?
Do I need an llms.txt file for my game's website?
What's the difference between llms.txt and robots.txt?
Does llms.txt help SEO?
What is llms-full.txt?
Should I use an llms.txt generator?
Sources
- Ahrefs — We analysed 137K sites: 97% of llms.txt files never get read
- Search Engine Journal — 97% of llms.txt files got no requests
- PPC Land — llms.txt adoption rises 8.8×, 97% of files get zero AI requests
- Google — Optimizing for generative AI features (updated 15 June 2026)
- Search Engine Roundtable — Google says no AI system currently uses llms.txt
- Search Engine Land — Google says normal SEO works for AI Overviews; llms.txt won't be used
