BigCommerce SEO for AI search: robots.txt, sitemap and llms.txt

How to view and edit robots.txt on BigCommerce

Every BigCommerce store ships with a robots.txt file. To see or change it:

1. In the control panel, go to Settings > Website.
2. Scroll down to the Search Engine Robots section.
3. Edit the rules and save.

You need the Manage Settings permission, plus Edit Channels if you run Multi-Storefront and want a separate file per storefront. The default file disallows account, cart, checkout, login, order status, search, wishlist and /admin/ pages for all crawlers. BigCommerce prints that default in its help article, so you can paste it back if an edit goes wrong.

When you save, BigCommerce merges your rules with its system defaults. If the same crawler appears in both, the groups are combined into one entry in the live file. A single bad line such as Disallow: / under User-agent: * blocks search engines and AI engines alike, so test the live file with the robots.txt checker after every change.

The AI crawler rules you cannot see

BigCommerce adds two sets of rules that never appear in the settings screen:

• Disallow rules for product listing URLs with filters applied (_bc_fsnf=1), to keep filtered pages out of search results.
• A group for "known high-volume and AI crawlers" with Crawl-delay: 10 and blocks on cart, checkout, account, admin and filter pages.

The list of crawlers in that group is not editable and only visible at yourstoredomain/robots.txt. The practical upshot: product and category pages stay open to AI bots such as GPTBot and ClaudeBot by default, and the ones on BigCommerce's list are simply slowed down. If you want to block a bot outright, BigCommerce points you to an infrastructure service such as Cloudflare, because some crawlers ignore robots.txt.

BigCommerce sitemap: xmlsitemap.php

The XML sitemap is generated for you at /xmlsitemap.php. It is split into separate sitemaps by page type, each holding up to 50,000 URLs, but you only submit the main URL. It includes products, categories, brands, web pages and blog pages that are marked visible. On Multi-Storefront, products and categories must also be assigned to that storefront's channel. Links update automatically when a URL changes, and BigCommerce suggests resubmitting only after big changes such as a new domain or thousands of new products.

Do not confuse it with /sitemap.php, a human-readable page of categories, brands and web pages for shoppers. On headless Catalyst storefronts, /xmlsitemap.php permanently redirects to /sitemap.xml, which Catalyst fetches from the default channel.

llms.txt and product structured data

BigCommerce does not generate llms.txt. Its help center suggests adding a web page with the type for raw HTML, pasting your plain-text content, and entering llms.txt in the Page URL field. Check the result with the llms.txt checker.

For product schema, the default Cornerstone theme switched from microdata to a JSON-LD script in version 6.2.0 (December 2021). The template outputs a Product with SKU, MPN, GTIN, brand, a description trimmed to 4,000 characters, the main image, an Offer with price or price range, condition and availability, and ratings plus reviews when reviews are shown. If your theme is older or third-party, view a product page's source and search for application/ld+json before assuming it is there.