AI search settings, platform by platform
Whether ChatGPT, Perplexity or Google's AI features can read your site starts with three files your platform controls: robots.txt, the XML sitemap and, optionally, llms.txt. Each guide shows where your platform keeps them, what you can change, and how it treats AI crawlers by default, from the platform's own docs.
Website builders and CMSs
- WordPressWhere WordPress serves its virtual robots.txt, how to edit it with or without Yoast or Rank Math, where /wp-sitemap.xml lives, and how to add llms.txt and schema.
- WixWix's Robots.txt Editor, its automatic /sitemap.xml, the auto-generated llms.txt file and preset JSON-LD markup, with the dashboard paths from Wix's own help center.
- SquarespaceSquarespace's shared robots.txt can't be edited, but its Crawlers settings can block AI bots. Where the sitemap lives, how the 7.1 llms.txt tab works and what schema it adds.
- WebflowWebflow's SEO tab covers most AEO plumbing: robots.txt rules, a Content-Signal header, an auto sitemap, llms.txt upload and a per-page Schema markup field. Paths and plan limits.
- FramerHow Framer handles AI search: auto robots.txt that allows AI crawlers, /sitemap.xml, Markdown responses for agents, llms.txt via static files, and JSON-LD in custom code.
- GoDaddy Website BuilderWhat GoDaddy's Websites + Marketing and Airo AI Builder do for SEO and AI search: the SEO tool path, the built-in sitemap, structured data, and what GoDaddy doesn't document.
- HubSpot CMSWhere HubSpot keeps robots.txt (SEO & Crawlers tab), how its /sitemap.xml works and why landing pages are left out, why llms.txt must be hosted elsewhere, and blog schema.
- Google SitesGoogle Sites has no sitemap file, no robots.txt editor and no way to add JSON-LD to the page head. What you can control instead: publishing, search visibility, Search Console and content.
- DrupalDrupal ships a robots.txt but no XML sitemap. How to edit robots.txt without Composer overwriting it, set up Simple XML Sitemap, add llms.txt and output JSON-LD.
- JoomlaJoomla ships robots.txt.dist, not a sitemap. What the default rules block, why updates never rename the file, which extensions add sitemaps and llms.txt, and Joomla 5 JSON-LD.
- BloggerBlogger builds robots.txt and sitemap.xml for you. Where the custom robots.txt setting is, what the default file contains, how to block AI crawlers, and why llms.txt is out of reach.
- WeeblyWeebly writes robots.txt and sitemap.xml for you and lets you hide pages, not edit rules. What that means for AI crawlers, llms.txt and structured data on a Weebly site.
- SubstackSubstack controls robots.txt and sitemaps for every publication. What the Block AI training toggle does, when sitemaps become available, and how to help AI cite a newsletter.
- ShowitShowit puts its page sitemap at /siteinfo.xml and the blog sitemap on WordPress. Where both live, how to submit them, what controls robots.txt and llms.txt, and how to add JSON-LD.
- LovableLovable apps are React single-page apps. How new TanStack Start apps render on the server, how older React + Vite apps pre-render for verified crawlers, and what other bots get.
- MintlifyMintlify generates llms.txt and llms-full.txt for every docs site automatically. Where they live, what goes in them, how to override them, and how robots.txt, sitemap and JSON-LD work.
Ecommerce platforms
- ShopifyHow Shopify generates robots.txt and how robots.txt.liquid overrides it, where /sitemap.xml lives, and what the Shopify-managed /llms.txt and /agents.md files contain.
- BigCommerceWhere BigCommerce keeps robots.txt and xmlsitemap.php, the hidden crawl delay it gives AI bots, how to publish llms.txt as a raw page, and the product schema Cornerstone outputs.
- WooCommerceHow a WooCommerce store serves robots.txt and wp-sitemap.xml, the Product JSON-LD WooCommerce outputs by default, and where llms.txt and AI shopping feeds fit in.
- Magento (Adobe Commerce)Magento (Adobe Commerce) robots.txt settings, how to generate and schedule sitemap.xml, adding it to robots.txt, AI crawler rules, and what product structured data ships by default.
- ShopwareShopware 6 robots.txt (built in since 6.7.1.0), sitemap.xml refresh settings, the JSON_LD_DATA flag that swaps microdata for JSON-LD, and how to handle AI crawlers and llms.txt.
- PrestaShopHow PrestaShop's Generate robots.txt button works, why it overwrites your edits, the gsitemap module's sitemap files, and the Product JSON-LD the Classic theme prints.
SEO plugins
- Yoast SEOHow Yoast SEO edits robots.txt, serves sitemap_index.xml, generates llms.txt for free since 25.3, blocks AI bots in Premium, and builds its schema graph, from Yoast's own docs.
- Rank MathRank Math SEO settings that affect AI search: the virtual robots.txt editor, sitemap_index.xml, the LLMS Txt module added in July 2025, and default JSON-LD schema types.
Frameworks
- Next.jsHow to generate robots.txt and sitemap.xml with Next.js App Router metadata files, serve /llms.txt, add JSON-LD, and keep content readable by AI crawlers.
- ReactWhy a client-only React SPA can look empty to AI crawlers, what the React team now recommends instead of Create React App, and how to add robots.txt, a sitemap and JSON-LD.
- AngularMake an Angular app readable by AI crawlers with @angular/ssr render modes, serve robots.txt and llms.txt from public/, and add schema.org JSON-LD that ships in the HTML.
- AstroSet up @astrojs/sitemap, write robots.txt as a static file or endpoint, generate /llms.txt, and add JSON-LD in Astro. Astro prerenders pages by default, which suits AI crawlers.
- NuxtNuxt renders on the server by default. How the Nuxt SEO modules handle robots.txt, sitemap.xml, schema.org and llms.txt, plus the settings that hide a Nuxt site from AI crawlers.
Hosting and CDN
- GitHub PagesGitHub Pages serves static files, which AI crawlers read well. How to add robots.txt, a sitemap with jekyll-sitemap, llms.txt and JSON-LD, and the project-site robots.txt trap.
- CloudflareHow Cloudflare's managed robots.txt, AI bot policies (Search, Agent, Training), AI Crawl Control and pay per crawl affect AI crawlers, and how a CDN default can hide you from AI search.
- VercelWhat Vercel does to crawlers: the automatic noindex header on previews, Deployment Protection, the Bot Protection and AI bots firewall rulesets, and why server rendering matters for AI.
Marketplaces
- EtsyEtsy SEO from Etsy's own seller handbook: how search matches and ranks listings, the 13 tags and attributes that matter, and how Etsy listings now reach shoppers inside ChatGPT.
- AmazonAmazon SEO for its AI assistant: what Rufus (now Alexa for Shopping) reads from your listing, the fields to fill, and why amazon.com's robots.txt keeps most outside AI crawlers out.