Does Drupal have a built-in XML sitemap?

No. Drupal core does not generate one. Install a contributed module such as Simple XML Sitemap, which serves the default sitemap at /sitemap.xml and is configured at admin/config/search/simplesitemap.

Why did my robots.txt changes disappear after a Drupal update?

robots.txt is a Composer scaffold file provided by drupal/core, so composer install or update can replace it. Add "[web-root]/robots.txt": false under extra.drupal-scaffold.file-mapping in your root composer.json, or use the append mode to add your own lines to core's file.

Does Simple XML Sitemap add the Sitemap line to robots.txt?

Drupal core's default robots.txt has no Sitemap line, so add one yourself, for example Sitemap: https://example.com/sitemap.xml. If you use the scaffold append mode, put it in your additions file.

Can Drupal serve llms.txt?

Yes, with the /llms.txt module. You author the content under Administration, Content, llms.txt. The module page says your web server must be configured to pass /llms.txt to Drupal, and gives an Nginx location block for that.

AI search readiness · Website builders and CMSs

Drupal SEO for AI search: robots.txt, XML sitemap and llms.txt

Drupal core ships a physical robots.txt in the web root, but no XML sitemap, no llms.txt and no JSON-LD. Each of those comes from a contributed module: Simple XML Sitemap for /sitemap.xml, the /llms.txt module for llms.txt, and Schema.org Metatag for structured data. The main trap is Composer scaffolding, which can put core's robots.txt back on every update.

Edit robots.txt: YesCore ships a real robots.txt file in the web root. Edit it directly (and stop Composer scaffold from overwriting it), or use the RobotsTxt module to edit it in the admin UI.
XML sitemap: With a plugin or appNot generated by core. The Simple XML Sitemap module serves one at /sitemap.xml by default. /sitemap.xml
llms.txt: With a plugin or appThe /llms.txt contributed module adds an editable /llms.txt endpoint for Drupal 10.3+ and 11.1+. Your web server must route the path to Drupal.
AI crawlers: No AI-specific setting documented in core. The default robots.txt has a single User-agent: * group, so every AI crawler gets the same rules as search engines until you add named groups.
Structured data: Core outputs no JSON-LD. The Schema.org Metatag module (which requires Metatag) adds JSON-LD in the page head, with defaults per content type and per-node overrides.

Where Drupal's robots.txt lives and how to edit it

Drupal core includes a real robots.txt in the web root. It is a Composer scaffold file: the original sits at core/assets/scaffold/files/robots.txt and is copied to [web-root]/robots.txt when Composer runs. The default file has one User-agent: * group. It allows CSS, JavaScript and images under /core/ and /profiles/, disallows those directories otherwise, and blocks paths such as /admin/, /user/login, /search/ and /node/add/. It has no Sitemap: line.

You have three ways to change it:

Append lines on every build. In your root composer.json, map "[web-root]/robots.txt" to { "append": "assets/my-robots-additions.txt" }. Drupal's scaffold docs use exactly this example. Good for adding a Sitemap: line or AI crawler groups while keeping core's defaults.
Take full ownership. Set "[web-root]/robots.txt": false under extra.drupal-scaffold.file-mapping, then edit the file by hand. Composer stops touching it.
Edit in the admin UI. The RobotsTxt module (Drupal 9.3 to 11) generates robots.txt dynamically, which suits multisite installs. Its project page says you must delete or rename the physical robots.txt for it to work, so combine it with option 2.

Drupal sitemap: Simple XML Sitemap

Core does not build an XML sitemap. Simple XML Sitemap (4.x, for Drupal 10.3 and 11) is the usual choice. After installing it:

The default sitemap is at /sitemap.xml.
Enable entity types at admin/config/search/simplesitemap/entities, then set inclusion per bundle on the bundle's edit page (for content types, admin/structure/types/manage/[type]).
Sitemaps regenerate on cron at the "Sitemap generation interval". drush simple-sitemap:rebuild-queue (alias ssr) requeues everything.
The simple_sitemap_engines submodule adds IndexNow, which the module page says Bing and Yandex support.

Add the Sitemap: line to robots.txt yourself, then submit the URL in Google Search Console and Bing Webmaster Tools.

llms.txt on Drupal

The /llms.txt module adds an endpoint you author under Administration > Content > llms.txt (/admin/content/llms-txt, with extra sections at /admin/content/llms-txt/sections). It supports Drupal 10.3+ and 11.1+. Because many server configs serve .txt files straight from disk, the module asks you to route /llms.txt to Drupal first. Check the result with the llms.txt checker. For what the file is and is not, see llms.txt.

AI crawlers and structured data

Core has no AI crawler setting. To treat bots differently, add named groups through the append file, for example blocking GPTBot (training) while keeping OAI-SearchBot (ChatGPT search) allowed. Remember that a named group replaces the * rules for that bot, so copy any Disallow lines it should still follow.

For JSON-LD, install Metatag plus Schema.org Metatag. It writes JSON-LD into the head and covers types such as Article, Organization, Product, FAQPage, Event and WebSite. Set defaults per content type with tokens, then override on single nodes when needed.

Check your Drupal site

Paste your domain to see which AI crawlers your live robots.txt lets in.

robots.txt checker

Test your robots.txt against every AI crawler

robots.txt generator

Allow AI search, block trainingStay citable in ChatGPT, Claude, Perplexity, Gemini and Copilot answers while opting out of model training.Allow everythingEvery crawler may read every public page. The default when there is no robots.txt.Block all AI crawlersKeep Google and Bing search, block every AI crawler. AI answers will rarely cite or recommend the site.

Sitemap URLPrivate paths (one per line)

robots.txt

# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training

User-agent: *
Allow: /

# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Questions

Does Drupal have a built-in XML sitemap?: No. Drupal core does not generate one. Install a contributed module such as Simple XML Sitemap, which serves the default sitemap at /sitemap.xml and is configured at admin/config/search/simplesitemap.
Why did my robots.txt changes disappear after a Drupal update?: robots.txt is a Composer scaffold file provided by drupal/core, so composer install or update can replace it. Add "[web-root]/robots.txt": false under extra.drupal-scaffold.file-mapping in your root composer.json, or use the append mode to add your own lines to core's file.
Does Simple XML Sitemap add the Sitemap line to robots.txt?: Drupal core's default robots.txt has no Sitemap line, so add one yourself, for example Sitemap: https://example.com/sitemap.xml. If you use the scaffold append mode, put it in your additions file.
Can Drupal serve llms.txt?: Yes, with the /llms.txt module. You author the content under Administration, Content, llms.txt. The module page says your web server must be configured to pass /llms.txt to Drupal, and gives an Nginx location block for that.

Other website builders and cmss

Sources

Checked against these pages on 2026-10-19. Operators change their crawlers; check the source before relying on a detail.