llms.txt
llms.txt is a proposed standard: a plain markdown file published at the root of your domain that gives large language models a curated, link-by-link map of the content you want them to read. It is a publishing convention rather than a rule, so nothing in the file obliges any model or crawler to obey it.
Seenby is not open yet, so anything on this page about what it does is what it is being built to do.
What llms.txt is
llms.txt is a proposed standard: a plain markdown file published at the root of your domain, at /llms.txt, that gives large language models a curated, link-by-link map of the content you want them to read. It was put forward in 2024 by Jeremy Howard of Answer.AI, and the reasoning behind it is simple. A model working with a limited context window cannot read your whole site, and HTML pages carry navigation, scripts and boilerplate that waste that window. So you hand it a short markdown index instead, and you choose what goes in it.
The file is meant to be readable by a person and parseable by a machine. The convention is deliberately narrow.
What the file looks like
- One H1 with the name of the site, product or project
- A blockquote of a sentence or two saying what it is
- Optional plain prose with any context a reader needs
- H2 sections, each holding a markdown list of links, with a colon and a short description after each one
- An optional section named Optional, holding pages a model can skip when context is tight
A companion convention, llms-full.txt, goes further: rather than linking to pages, it concatenates the full markdown of your documentation into a single file. Many documentation platforms also publish a .md version of each page at the same URL, so a model can fetch clean text instead of rendered HTML.
What it does and does not do
The honest position is that llms.txt is an offer, not an instruction. We know of no public commitment from a major AI provider to read /llms.txt as part of its retrieval or training pipeline, and the file has no mechanism to require one. Anyone telling you it guarantees inclusion in AI answers is describing an outcome the format cannot deliver.
What it does do is cheap and low-risk. It costs an afternoon, it creates no duplicate content problem, it gives agents and internal tools a clean entry point, and it forces a useful editorial exercise: naming the ten or twenty pages you would actually want an answer engine to quote. That exercise tends to be more valuable than the file.
Two cautions. First, llms.txt is a supply-side guess about demand-side behaviour, so treat it as a hypothesis you test, not a fix you ship. Second, it is not an access control. If you want to restrict crawling, that argument belongs in robots.txt, and robots.txt is itself voluntary.
llms.txt compared with the files it gets confused with
| File | What it is for | Who it is aimed at | Status |
|---|---|---|---|
| robots.txt | Stating which paths a crawler should not fetch | Search and AI crawlers | Long-established voluntary convention, widely implemented |
| sitemap.xml | Listing every indexable URL with change and priority hints | Search engine indexers | Long-established voluntary convention, widely implemented |
| llms.txt | Pointing a model at a curated subset of pages, in markdown, with descriptions | Large language models and AI agents | Proposed convention, adoption by providers not publicly committed |
| llms-full.txt | Serving the full text of that content as one markdown file | Large language models with room to ingest it | Companion convention to llms.txt, same uncertainty |
How to tell whether it changed anything
Publishing the file is the easy half. Reading the result is the hard half, because the answers you are trying to influence are non-deterministic. Ask the same question twice and the wording moves. Ask it after a model update and the substance can move.
So the test has to be structural rather than anecdotal. Fix a prompt set before you publish, run it repeatedly, record which brands get named and in what order, then keep running the identical set afterwards. A single spot check in a chat window proves nothing either way. And when a number does move, the first question is whether the model version changed underneath it, because a model update can shift a score without anything changing in the market.
If your real question is which changes actually influence what a model says, that is a different and larger topic. Our guide on how to rank in ChatGPT covers what moves a brand into an answer, and where llms.txt sits in that order of priority.
Related terms
- Generative engine optimization (GEO): the practice of shaping content and off-site evidence so generative models describe and recommend you accurately. llms.txt is one tactic inside it.
- Answer engine optimization (AEO): the same discipline framed around answer surfaces such as AI Overviews and Perplexity, where the goal is being the cited source rather than the ranked link.
- Share of AI voice: the proportion of answers in a fixed prompt set that name your brand rather than a named competitor. It is the metric that tells you whether a change like llms.txt did anything.
- Citation: a linked source an answer engine attributes a statement to. Being cited and being mentioned are different outcomes and worth counting separately.
- Retrieval augmented generation (RAG): the pattern where a model fetches documents at query time before answering. It is the mechanism llms.txt hopes to feed.
- Model drift: the shift in outputs caused by a provider updating a model, as distinct from a real change in how the market talks about you.
- robots.txt: the crawl directive file. Related in spirit, opposite in purpose: one restricts, the other invites.
Questions
Does llms.txt actually work?
Unproven. The format is a proposal, and we know of no public commitment from a major AI provider to fetch and use /llms.txt. Treat it as a low-cost experiment you measure with a fixed prompt set, not as a mechanism with a promised outcome.
Do AI crawlers respect llms.txt?
There is no enforcement in the format, so respect is entirely voluntary, and providers publish little about it. If your aim is to restrict crawling rather than invite reading, robots.txt is the file you want, and that is voluntary too.
Where does the file go?
At the root of your domain, served as plain markdown with a text content type. The companion file, when you publish one, goes at /llms-full.txt.
Is llms.txt the same as a sitemap?
No. A sitemap lists every indexable URL for an indexer. llms.txt lists a curated subset in markdown, with a short description of each one, on the assumption that the reader has limited room and needs guidance about what matters.
What should I put in it?
The pages you would want an answer engine to quote back: product and pricing pages, documentation, definitional explainers, and anything that states plainly what you do and who you serve. Leave out navigation, marketing filler and anything you would not want restated in an answer.
Will adding it change my Google rankings?
Nothing about the format suggests it would. It is aimed at language models reading text, not at classical ranking systems, so judge it on how models describe you rather than on search positions.
How would I know if it made a difference?
Run the same prompt set before and after, count mentions and positions across several weeks, and check the model version behind any move. Week-over-week shifts can reflect a model update rather than a real change in the market, so a single week is not evidence.
Sources
Related
- How to rank in ChatGPT: what actually moves a brand into an answer
ChatGPT has no ranking table, so the real question is what makes it name your brand in an answer.
- Generative engine optimization (GEO)
Generative engine optimization is the practice of shaping how AI answer engines describe and recommend your brand.
- LLM SEO
A definition of LLM SEO, the practice of shaping how large language models describe and recommend a brand, with the related terms it gets confused with and the measurement problem underneath it.
