# llms.txt: what it is, what it is not, and when it actually helps > llms.txt is a proposal, not a standard. What the format specifies, how files fail, and how it relates to llms-full.txt and sitemap.md. - Source: /learn/llms-txt-guide - Updated: 2026-10-08 - Reading time: 7 minutes - Tags: llms.txt, agent readiness, discovery --- `/llms.txt` is the closest thing agent readiness has to a shared convention, and it is still only a proposal. It is also the cheapest single signal a site can add. Both halves matter, and most writing about the file quietly drops one of them. ## What the file actually specifies The proposal at [llmstxt.org](https://llmstxt.org/) asks for a markdown file at a conventional path: `/llms.txt` at the root, or inside any subpath, where it describes the pages under that path. The most specific file wins. The page dates the original proposal to September 3, 2024 and the current v2 revision to August 10, 2026. A conforming file has a fixed order: - an optional byte-order mark; - an H1 naming the project or site, the only required section; - a blockquote summary carrying the facts needed to interpret the rest; - optional prose sections with background and usage notes; - zero or more H2 sections, each a markdown list of `[name](url)` links, optionally followed by `:` and notes. That is the whole contract. v2 adds two things worth shipping: a clean markdown version of each page an agent might need, at the same URL with `.md` appended or substituted (`index.md` for directory URLs), and link relations advertising it — `rel="alternate" type="text/markdown"` for the markdown page, `rel="describedby"` for the covering llms.txt. Both work as HTML `` elements or an HTTP `Link:` header, and the header ships from server or CDN config without touching any page. The proposal wants the file readable by people and models but parseable by ordinary code. If a regex cannot find your links, a scanner cannot either. One loophole is worth naming: because the H1 is the only required section, a file containing nothing but an H1 conforms to the letter of the proposal and is useless. Conformance and usefulness are different tests. ## What it is not **It is not a standard.** There is no RFC and no W3C work item. The proposal lives as an informal overview in a [GitHub repository](https://github.com/AnswerDotAI/llms-txt) with an open issue list, which means no conformance body and no version negotiation. The proposal itself rejects the well-known URI approach, RFC 8615, because well-known paths exist only at the origin root while llms.txt describes whatever path it sits at. **Nobody enforces it.** No crawler, model vendor, or platform is required to fetch the file, parse it, or act on what it says. Lighthouse's agentic browsing audit shows the posture: it flags a page when fetching `/llms.txt` returns a server error, but reports it out of scope on a 404, because [serving the file is optional](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). The tooling is agreeing, in public, that omission is legal. **It is not a replacement for robots.txt.** [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) defines robots.txt as rules a crawler is requested to honour, and states that those rules "are not a form of access authorization." llms.txt has the opposite job: it invites reading and curates it. Access rules and reading guidance are separate concerns, and adding llms.txt does not change what any crawler may fetch. **It is not a sitemap, and it is not a ranking factor.** The proposal makes the sitemap argument itself: sitemap.xml lists everything indexable for search engines, usually omits markdown variants, cannot point at external resources, and is far too large for a context window. On ranking, nothing in the proposal claims an effect, and we have not found a search engine that documents llms.txt as a ranking input. That is absence of evidence rather than proof, but treat "add llms.txt for SEO" as unsupported. The honest case is agent legibility, not traffic. **It is not a guarantee that any model reads it.** Whether a given agent or crawler fetches the file is undocumented, uneven, and changes without notice. You cannot observe it from outside, and we do not claim to. ## Adoption: a trend signal, not a market share Among the sites that opted into being measured, roughly 46% served a `/llms.txt`, and those files averaged 37 out of 100 on structure and content quality. The caveat matters more than the figures. The corpus is self-selected: sites arrive because somebody decided agent readiness was worth checking, which correlates with having already done part of the work and probably inflates the rate relative to the open web. It is not a market share, it does not generalise beyond the corpus, and we have no error bars that would let us pretend otherwise. The [methodology](/methodology) page describes how the corpus is assembled. Three things can be said without inventing numbers. The proposal's author writes that "thousands of sites" publish one, which is the proposer's characterisation rather than a census. Directories exist, linked from the proposal, but a directory is a list of submissions, not a sample. And many files are generated rather than written: platforms and CMS plugins now emit one automatically, and Mintlify documents generation for every site it hosts. Presence no longer implies intent, and it never implied quality. ## What a good one contains ``` # Example Inc > Example Inc runs a payments API for marketplaces. This file lists the pages an agent needs, with a note on each. US-based. Support: support@example.com. API access uses OAuth 2.0 tokens; links below are markdown versions unless noted. ## Docs - [Quickstart](https://example.com/docs/quickstart.md): Install the CLI and make a first API call - [API reference](https://example.com/docs/api.md): Every endpoint, with request and response examples - [Authentication](https://example.com/docs/auth.md): OAuth 2.0 flows and token lifetimes ## Company - [Pricing](https://example.com/pricing.md): Plans, limits and overage rates - [Full context](https://example.com/llms-full.txt): Every page above as one document ## Optional - [Changelog](https://example.com/changelog.md): Release history, for version-specific questions only ``` The properties that make this work are structural. The blockquote carries identity and the facts that make the rest interpretable; the prose line states constraints an agent cannot infer — currency, support route, auth model. Links point at markdown variants where they exist, not at HTML that needs conversion. Sections are grouped by task rather than by your navigation, and `Optional` marks context an agent can skip. Two habits matter as much as the content: generate the file at build time so it cannot drift, and test it by giving an agent only the file and asking questions about your product. A `Link: ; rel="describedby"` header then lets clients find it without guessing the path. ## How a bad one fails ### An H1 with no links Conformant and worthless. With no H2 list the file cannot lead anywhere, which is why the [llms-txt check](/checks/llms-txt) is structural: it scores what is verifiable rather than what sounds good. ### A marketing page pasted in Adjectives, no paths, no constraints. The problem is not tone: an agent cannot act on "leading platform for modern teams," and every sentence that is not a link or a fact spends context. ### Stale links Worse than no file: it spends an agent's request budget on a 404 and teaches it that your guidance is unreliable. Drift comes from hand maintenance; generate the file and link-check it in CI. ### HTML served with a 200 The classic failure is a single-page app whose catch-all rewrite answers `/llms.txt` with the application shell, `Content-Type: text/html`, status 200. The same rewrite usually means every unknown path answers 200 with the same document, which is what the [spa-fallback check](/checks/spa-fallback) measures. Serve `text/plain` or `text/markdown` and let a missing file 404 honestly. A quieter variant is an llms.txt behind authentication, which no unauthenticated agent can read. ## llms-full.txt and sitemap.md: three files, three jobs **`/llms.txt`** is an index: small enough to fit in context, deferring detail to links. It is the default, and for most sites the only one of the three that matters. **`/llms-full.txt`** is the corpus in one file. Be precise about its status: the llmstxt.org proposal, as published when we checked, defines llms.txt and per-page markdown variants, not llms-full.txt. The filename is a de facto convention, documented by platform vendors such as [Mintlify](https://www.mintlify.com/docs/ai/llmstxt), that concatenates key pages into one document — title, source URL, then full markdown per page. It is right when the corpus is bounded: a documentation set, an API reference, a policy corpus. It is wrong for a large catalogue, where it either exceeds a context window or omits most of the site. The [llms-full check](/checks/llms-full) requires that llms.txt links to it, because an unadvertised file is rarely fetched. **`/sitemap.md`** is a markdown map of the site, also absent from the proposal; we treat it as an emerging convention and probe that exact path. Where llms.txt is a curated reading list with notes, sitemap.md is a flat index of absolute markdown links, closer in spirit to sitemap.xml. If you publish only one, publish llms.txt. The [sitemap-md check](/checks/sitemap-md) carries a low weight because adoption is still thin. ## It is optional, and it is the cheapest signal you can add llms.txt is optional in the strict sense. Lighthouse reports it out of scope on a 404, no vendor requires it, and a site can be fully agent-ready without it. Server-rendered pages that answer each URL with its own content, a sitemap.xml that matches reality, robots.txt that states what may be fetched, and structured data will take an agent further than a good text file on a site it cannot parse. If you must choose where to spend a sprint, fix the rendering first. The cost asymmetry is extreme: one static file, generated at build time, reviewed once. It is the only signal in this area with a published format, existing tooling, and a path an agent can guess from the domain alone. That is why our rubric scores it at the essential tier with a heavy weight while still calling it optional: optional for the web, counted by us. ## Verify yours in a minute ``` curl -sS -D- -o /tmp/llms.txt https://example.com/llms.txt | head -5 curl -sS https://example.com/llms.txt | head -30 ``` Check four things: status 200, `Content-Type: text/plain` or `text/markdown`, an H1 followed by a blockquote, and at least one H2 section whose items are `[name](url)` links. Then fetch three links by hand. ## What we cannot measure Our check can confirm that a file exists, parses, and covers the pages it claims to. It cannot tell you whether an agent read it, whether that changed an answer, or whether vendors will keep looking next quarter. Nobody publishes that data, and we will not manufacture it. So the defensible position is the modest one: llms.txt is a cheap, well-specified way to describe your site to an agent that might read it. It is worth an afternoon. It is not worth a rewrite, and not a substitute for the unglamorous work — real URLs, real content, honest status codes — that makes a site legible in the first place.