Where do I edit robots.txt in Magento 2?

In the Admin, go to Content > Design > Configuration, click Edit on the Global row, and expand Search Engine Robots. Enter rules in the custom instructions box and click Save Configuration.

How do I add my sitemap to robots.txt in Magento?

Go to Stores > Settings > Configuration > Catalog > XML Sitemap and set Enable Submission to Robots.txt to Yes in Search Engine Submission Settings.

Why don't my robots.txt changes show on Adobe Commerce on Cloud?

On Cloud, robots.txt is generated on demand from the database, and custom rules only apply where search engine indexing is enabled, which is Production only. Lower environments keep the default output.

Does Magento block AI crawlers by default?

Not specifically. The default rules apply to all crawlers and block system paths such as /checkout/, /customer/ and /catalog/, so product URLs stay open unless you add rules for named AI bots.

AI search readiness · Ecommerce platforms

Magento SEO for AI search: robots.txt, sitemap.xml and schema

Magento Open Source and Adobe Commerce generate robots.txt from the Admin: you edit it under Content > Design > Configuration in the Search Engine Robots section. You create the XML sitemap yourself under Marketing > SEO & Search > Site Map and can have Commerce add it to robots.txt. There is no llms.txt feature or AI crawler toggle, and schema.org markup is added to the product template by default.

Edit robots.txt: YesContent > Design > Configuration > Global > Search Engine Robots: pick Default Robots, add directives in the custom instructions box, or click Reset to Default.
XML sitemap: YesYou add it under Marketing > SEO & Search > Site Map and choose the filename and path. Schedule it under Stores > Settings > Configuration > Catalog > XML Sitemap. Cloud projects must write to /media/. /sitemap.xml
llms.txt: NoNo llms.txt generator in Adobe's Commerce docs. You would have to serve a static file yourself or use an extension.
AI crawlers: No AI-specific setting. Add user-agent groups for AI bots in the custom robots.txt instructions. Adobe lists a Catalog Agent that makes product data readable to AI crawlers as coming soon.
Structured data: Adobe says schema.org structured data markup is added to the store's product template by default. Themes and extensions often change or duplicate it, so check the rendered product page.

How to edit robots.txt in Magento and Adobe Commerce

Commerce builds robots.txt from Admin settings, so on most installs you never touch a file:

In the Admin sidebar, go to Content > Design > Configuration.
In the Global row, click Edit.
Expand Search Engine Robots.
Set Default Robots (INDEX, FOLLOW for a live store).
Add rules in Edit Custom instruction of robots.txt File.
Click Save Configuration. Reset to Default restores Adobe's rules.

The default instructions disallow internal paths such as /app/, /lib/, /var/, /checkout/, /customer/, /catalog/, /review/ and session ID parameters. On Adobe Commerce on Cloud, robots.txt is generated on demand and stored in the database, and it is reachable at both /robots.txt and /robots. Custom rules only take effect in Production, because search engine indexing cannot be enabled in lower environments.

Generating and submitting sitemap.xml

Magento does not create a sitemap until you add one:

Go to Marketing > SEO & Search > Site Map and click Add Site Map.
Enter a filename such as sitemap.xml and a writable path (/ for the root).
Click Save & Generate.

Under Stores > Settings > Configuration > Catalog > XML Sitemap you set frequency and priority for categories, products, CMS pages and images, and in Generation Settings a daily, weekly or monthly schedule. File limits default to 50,000 URLs and 10,485,760 bytes per file. Set Enable Submission to Robots.txt to Yes so Commerce adds the XML sitemap to robots.txt. On Cloud the root is read-only, so generate the file under /media/.

AI crawlers and llms.txt

There is no AI bot toggle in Commerce. If you want to treat AI crawlers differently, add groups by name in the custom instructions box, for example blocking GPTBot for training while leaving OAI-SearchBot and PerplexityBot allowed. Then confirm the live result with the robots.txt checker, because a CDN rule in front of the store can still block bots that robots.txt allows.

Adobe's docs describe no llms.txt generator. Adobe has announced a Catalog Agent that would add a machine-readable product layer for AI crawlers without changing the storefront, but its documentation marks it as coming soon.

Product structured data

Adobe's meta data guide says schema.org structured data markup is added to the product template by default. The product page also takes a Meta Title (Adobe suggests under 70 characters) and Meta Description (150 to 160 characters). Because custom themes and SEO extensions often replace or add markup, check a live product page in Google's Rich Results Test and make sure only one Product entity appears, with price, availability and identifiers that match the page.

Check your Magento (Adobe Commerce) site

Paste your domain to see which AI crawlers your live robots.txt lets in.

robots.txt checker

Test your robots.txt against every AI crawler

robots.txt generator

Allow AI search, block trainingStay citable in ChatGPT, Claude, Perplexity, Gemini and Copilot answers while opting out of model training.Allow everythingEvery crawler may read every public page. The default when there is no robots.txt.Block all AI crawlersKeep Google and Bing search, block every AI crawler. AI answers will rarely cite or recommend the site.

Sitemap URLPrivate paths (one per line)

robots.txt

# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training

User-agent: *
Allow: /

# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Questions

Where do I edit robots.txt in Magento 2?: In the Admin, go to Content > Design > Configuration, click Edit on the Global row, and expand Search Engine Robots. Enter rules in the custom instructions box and click Save Configuration.
How do I add my sitemap to robots.txt in Magento?: Go to Stores > Settings > Configuration > Catalog > XML Sitemap and set Enable Submission to Robots.txt to Yes in Search Engine Submission Settings.
Why don't my robots.txt changes show on Adobe Commerce on Cloud?: On Cloud, robots.txt is generated on demand from the database, and custom rules only apply where search engine indexing is enabled, which is Production only. Lower environments keep the default output.
Does Magento block AI crawlers by default?: Not specifically. The default rules apply to all crawlers and block system paths such as /checkout/, /customer/ and /catalog/, so product URLs stay open unless you add rules for named AI bots.

Other ecommerce platforms

Sources

Checked against these pages on 2026-10-19. Operators change their crawlers; check the source before relying on a detail.