Does Mintlify generate llms.txt automatically?

Yes. Every Mintlify site serves /llms.txt and /llms-full.txt, plus copies at /.well-known/llms.txt and /.well-known/llms-full.txt. Mintlify describes the files as always up to date with zero maintenance.

How do I customize Mintlify's llms.txt?

Add a file named llms.txt (or llms-full.txt) at the root of your docs project. It replaces the generated file of the same name. It must include an H1 site title. Delete the file to go back to the automatic version.

What is the difference between llms.txt and llms-full.txt on Mintlify?

llms.txt is an index: site title, description, and a link to each page's Markdown version with a short description. llms-full.txt combines the whole documentation site into one file, with each page's title, URL, description and full Markdown.

Why does my Mintlify llms.txt return 404?

The Don't index project setting on the dashboard's General page returns 404 for llms.txt, adds noindex to every page and empties the sitemap. Sites behind full authentication also require login for both files.

AI search readiness · Website builders and CMSs

Mintlify llms.txt, llms-full.txt, robots.txt and sitemap for AI search

Mintlify is one of the few platforms where llms.txt needs no work: every docs site gets an automatic /llms.txt and /llms-full.txt, kept current on each deploy, with a root-level file to override either one. robots.txt and sitemap.xml are generated the same way, and indexable pages carry a JSON-LD graph. The settings that matter live in docs.json and page frontmatter.

Edit robots.txt: YesGenerated automatically, with default Content-Signal lines. Add robots.txt at the project root to override it; the custom file is served as-is.
XML sitemap: YesGenerated automatically from the docs.json navigation; hidden pages are left out. Add sitemap.xml at the project root to override it. /sitemap.xml
llms.txt: YesBuilt in. /llms.txt and /llms-full.txt (also under /.well-known/) are generated and kept up to date. A root-level file of the same name overrides each.
AI crawlers: The generated robots.txt includes Content-Signal: ai-train=yes, search=yes, ai-input=yes. A custom robots.txt drops those lines. Responses also carry an X-Llms-Txt header pointing to /llms.txt.
Structured data: Indexable pages get a JSON-LD @graph with Organization, WebSite, WebPage, BreadcrumbList and TechArticle or APIReference. Configure the publisher with seo.organization in docs.json; noindex pages get none.

Mintlify llms.txt and llms-full.txt

Mintlify generates both files for every docs site and regenerates them as content changes:

File	Paths	What it contains
llms.txt	`/llms.txt`, `/.well-known/llms.txt`	H1 site title, the `description` from `docs.json`, optional agent instructions from `markdown.instructions`, a link to each page's `.md` version with its description, API spec links and an Optional section
llms-full.txt	`/llms-full.txt`, `/.well-known/llms-full.txt`	Every page's title, source URL, description and full Markdown in one file

A few details decide what ends up in them:

Order follows your docs.json navigation. Pages outside navigation but included through seo.indexing: "all" are listed last, alphabetically.
Hidden and noindex pages are left out unless seo.indexing is "all".
Descriptions come from each page's frontmatter description, or the first paragraph cut at 300 characters.
Large sites: if the index passes 100,000 characters, Mintlify moves page groups into generated indexes under /_llms/. If you run Mintlify behind a reverse proxy or path allowlist, forward /_llms/ too.
Audience blocks: content marked for="agents" goes into llms-full.txt; content marked for="humans" stays out.

To customize, add llms.txt or llms-full.txt at the project root. It overrides the generated file and must include an H1 title. Responses also send a Link header and X-Llms-Txt: /llms.txt so agents can find it. Confirm what is live with the llms.txt checker, and see llms.txt for what the format can and cannot do.

robots.txt and sitemap.xml

Mintlify generates robots.txt with Content-Signal directives (ai-train=yes, search=yes, ai-input=yes). A root-level robots.txt replaces it and is served exactly as written, without those lines, so copy them in if you still want them. This is where you would block a training crawler such as GPTBot while leaving AI search crawlers like Claude-SearchBot allowed.

The XML sitemap at /sitemap.xml lists pages in your docs.json navigation and skips hidden pages; a root-level sitemap.xml overrides it. On fully authenticated sites it lists no pages.

Indexing controls and structured data

noindex: true in frontmatter removes one page; hidden: true implies it.
seo.metatags.robots: "noindex" in docs.json applies site-wide, and seo.paths scopes meta tags to folders.
Don't index project on the dashboard's General page adds noindex, nofollow everywhere, empties the sitemap and returns 404 for llms.txt. It does not touch a custom robots.txt.

Indexable pages also get a JSON-LD @graph with Organization, WebSite, WebPage, BreadcrumbList and TechArticle (or APIReference for API pages). Set your company's name, logo and sameAs profiles under seo.organization so AI engines connect the docs to the right entity.

Check your Mintlify site

Paste your domain to see which AI crawlers your live robots.txt lets in.

robots.txt checker

Test your robots.txt against every AI crawler

robots.txt generator

Allow AI search, block trainingStay citable in ChatGPT, Claude, Perplexity, Gemini and Copilot answers while opting out of model training.Allow everythingEvery crawler may read every public page. The default when there is no robots.txt.Block all AI crawlersKeep Google and Bing search, block every AI crawler. AI answers will rarely cite or recommend the site.

Sitemap URLPrivate paths (one per line)

robots.txt

# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training

User-agent: *
Allow: /

# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Questions

Does Mintlify generate llms.txt automatically?: Yes. Every Mintlify site serves /llms.txt and /llms-full.txt, plus copies at /.well-known/llms.txt and /.well-known/llms-full.txt. Mintlify describes the files as always up to date with zero maintenance.
How do I customize Mintlify's llms.txt?: Add a file named llms.txt (or llms-full.txt) at the root of your docs project. It replaces the generated file of the same name. It must include an H1 site title. Delete the file to go back to the automatic version.
What is the difference between llms.txt and llms-full.txt on Mintlify?: llms.txt is an index: site title, description, and a link to each page's Markdown version with a short description. llms-full.txt combines the whole documentation site into one file, with each page's title, URL, description and full Markdown.
Why does my Mintlify llms.txt return 404?: The Don't index project setting on the dashboard's General page returns 404 for llms.txt, adds noindex to every page and empties the sitemap. Sites behind full authentication also require login for both files.

Other website builders and cmss

Sources

Checked against these pages on 2026-10-19. Operators change their crawlers; check the source before relying on a detail.