AI search glossary
What is llms.txt? The format, an example and who uses it
llms.txt is a proposed standard, introduced by Jeremy Howard in September 2024, for a Markdown file at /llms.txt that gives language models and AI agents a short summary of a site and a curated list of links to its most useful pages.
Also called: llms txt, /llms.txt
Why it matters for AI search
Websites are built for browsers: navigation, scripts and layout make it hard for an agent with a limited context window to find the facts that matter. An llms.txt file hands the agent a clean map in plain Markdown. It does not replace crawlable pages or robots.txt, and no engine promises extra citations for having one, but it lowers the cost of reading your site correctly.
What the proposal says
Jeremy Howard proposed /llms.txt on September 3, 2024. The specification at llmstxt.org describes a Markdown file with sections in a fixed order:
- An H1 with the site or project name. This is the only required section.
- A blockquote with a short summary and the key facts needed to understand the rest.
- Optional detail in any Markdown except headings.
- H2 sections of links, each a Markdown link with an optional note after a colon.
- An "Optional" H2 section for secondary links an agent can skip when it needs less context.
The current version of the proposal also lets a file sit under a subpath and cover the URLs beneath it, with agents using the most specific file. It suggests serving a clean Markdown copy of important pages at the same URL with .md added, and lets pages point to their llms.txt with rel="describedby".
A minimal example
# Acme Coffee
> Small-batch coffee roaster in Portland, Oregon. Ships whole-bean
> coffee in the US. Subscriptions from one bag a month.
## Shop
- [Coffee range](https://acme.example/coffee.md): origins, roast levels, prices
- [Subscriptions](https://acme.example/subscribe.md): plans and how to pause
## Optional
- [Our story](https://acme.example/about.md)
Who uses it, and what it does not do
Developer documentation sites adopted it first. Anthropic's Claude docs, OpenAI's developer site and Perplexity's docs all serve an llms.txt file. For a store or service business the file is cheap to add, but keep expectations realistic:
- It is not access control. Use robots.txt to allow or block crawlers, and check it with the robots.txt checker.
- It is not a ranking signal any major engine has documented. Google states its AI features need nothing beyond normal indexing.
- It cannot fix thin pages. The links should point at content that already answers buyer questions.
How to add one
Write it by hand, or generate it from your sitemap and edit the result. Shopify merchants can use the llms.txt generator for Shopify. Then validate the structure with the llms.txt checker. Keep it next to your XML sitemap: the sitemap lists every URL for search crawlers, while llms.txt picks the few that explain your business to an AI agent.
Questions
- Is llms.txt an official web standard?
- No. It is a community proposal published at llmstxt.org, not a W3C or IETF standard. Sites adopt it voluntarily, and AI companies decide for themselves whether to read it.
- Does llms.txt block or allow AI crawlers?
- No. Access control stays in robots.txt. llms.txt only describes and points to content; it has no allow or disallow rules.
- Do I need llms.txt to appear in Google AI Overviews?
- No. Google says there are no additional requirements or special optimizations to appear in AI Overviews or AI Mode beyond being indexed and eligible for a snippet.
- What is the only required part of an llms.txt file?
- An H1 heading with the name of the project or site. The summary blockquote, detail paragraphs and H2 link lists are all optional but recommended.