Astro SEO and sitemap: @astrojs/sitemap, robots.txt and llms.txt

Setting up @astrojs/sitemap

The official sitemap integration installs with one command:

npx astro add sitemap

It needs your production URL. Set site in astro.config.mjs, and per the docs it must start with http:// or https://:

import { defineConfig } from 'astro/config';
import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://www.example.com',
  integrations: [sitemap()],
});

On build, the integration writes sitemap-index.xml and sitemap-0.xml; very large sites get extra numbered files. Submit /sitemap-index.xml, not /sitemap.xml, which Astro does not create. Use the filter option to drop pages: it receives each page's full URL, and returning false leaves the page out. That is the place to exclude thank-you pages, drafts and internal tools.

One limit to know: the integration covers dynamic routes generated by getStaticPaths(), but it cannot create entries for dynamic routes rendered on demand in SSR mode. If you serve those, generate a separate sitemap for them with an endpoint and reference both in robots.txt.

robots.txt and llms.txt

The sitemap integration does not write robots.txt. The Astro docs give two options: a static public/robots.txt, or an endpoint at src/pages/robots.txt.ts. A static version looks like this:

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap-index.xml

Endpoints are the better fit for llms.txt. Astro's endpoint docs show that a file such as src/pages/builtwith.json.ts exporting a GET function produces /builtwith.json at build time. In the same way, src/pages/llms.txt.ts can read your content collections and return a Markdown Response listing your key pages, so the file updates on every build. Validate it with the llms.txt checker.

If you add AI crawler rules, be precise about which token does what. Blocking GPTBot opts out of OpenAI training; OAI-SearchBot is the one that decides whether ChatGPT search can show you.

JSON-LD in Astro

Astro has no structured data helper, but its set:html directive makes JSON-LD simple. The directives reference shows it on a script tag:

<script type="application/ld+json" set:html={JSON.stringify(schema)} />

Astro treats any script with an attribute other than src as is:inline, so the JSON is written into the HTML as-is rather than bundled. The docs also warn that set:html does not escape its value, so do not pass untrusted user input. Put Organization JSON-LD in your base layout and page-specific types in each page or content collection layout.

Rendering and AI crawlers

Astro's docs state that pages, routes and endpoints are prerendered at build time by default. You opt individual routes into on-demand rendering with export const prerender = false and a server adapter. Either way the HTML is produced on the server, and client-side islands only add interactivity on top. That matters because Vercel's December 2024 analysis found that the major AI crawlers it tested do not render JavaScript. The one thing to avoid is loading core text inside a client-only island (client:only), which leaves it out of the HTML. Check the live site with the robots.txt checker after deploying.