WordPress robots.txt, sitemap and llms.txt for AI search

How WordPress serves robots.txt

A fresh WordPress install has no robots.txt on disk. When a crawler requests /robots.txt, WordPress runs the do_robots() function and prints a virtual file. Its default output is short:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The Sitemap line is added by the core sitemap feature introduced in WordPress 5.5. Nothing in the default file mentions AI crawlers, so GPTBot, ClaudeBot, PerplexityBot and the rest are allowed to crawl everything outside /wp-admin/.

One historical trap: before WordPress 5.3, ticking Settings > Reading > Search engine visibility ("Discourage search engines from indexing this site") changed the virtual file to Disallow: /. Since 5.3 it adds a noindex robots meta tag instead, and it switches off /wp-sitemap.xml. Either way it is the first thing to check when a live site is missing from search.

How to edit robots.txt in WordPress

There are three ways, and they interact. A physical file always wins.

1. Upload a physical file. Create robots.txt in the site root (the folder with wp-config.php) by SFTP or your host's file manager. The web server serves it directly and WordPress's virtual file is never generated.
2. Use an SEO plugin editor. In Yoast SEO go to Yoast SEO > Tools > File editor > Create robots.txt file. The menu is hidden when file editing is disabled, and the file must be writable. In Rank Math turn on Advanced Mode, then go to Rank Math SEO > General Settings > Edit robots.txt. Rank Math edits the virtual file, so if a physical robots.txt exists its changes do nothing until you delete that file.
3. Use the robots_txt filter (available since WordPress 3.0). It receives the generated output and a flag for whether the site is public. This adds a rule without touching the defaults:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    return $output;
}, 10, 2 );

The filter only runs when there is no physical file. After any change, load yourdomain.com/robots.txt in a private window and run it through the robots.txt checker; page caches and CDNs sometimes keep serving the old version.

The WordPress sitemap

Core publishes a sitemap index at /wp-sitemap.xml. It covers public post types, public taxonomies, author archives and the homepage, with up to 2,000 URLs per sub-sitemap. Developers can turn it off with the wp_sitemaps_enabled filter or drop a post type with wp_sitemaps_post_types.

SEO plugins replace it. With Yoast's XML sitemaps toggle on (Yoast SEO > Settings > Site features), or with Rank Math's sitemap module, the index is /sitemap_index.xml. Yoast's spec also redirects /sitemap.xml to its index. Submit whichever URL your robots.txt lists in Google Search Console and Bing Webmaster Tools; a WordPress site that switched plugins often still has the old sitemap submitted. More on how sitemaps work: XML sitemap.

llms.txt and structured data

Core has no llms.txt feature. Yoast SEO generates one under Settings > Site features > AI tools > LLMS.txt (not available on multisite). Rank Math has an LLMS Txt module configured at General Settings > Edit llms.txt, which lists chosen post types and taxonomies up to a default limit of 100 entries and skips noindexed posts. Both plugins call llms.txt a proposal with limited adoption. See what llms.txt is before you spend time on it, and test the result with the llms.txt checker.

For schema, Yoast outputs a single JSON-LD @graph with Organization or Person, WebSite and WebPage nodes linked by @id; its blog post example adds Article and BreadcrumbList. Rank Math and many themes do the same job, so run one schema source, not two, or you will ship duplicate entities.