Next.js SEO for AI search: robots.ts, sitemap.ts and llms.txt

robots.txt and sitemap.xml with App Router metadata files

Next.js has dedicated file conventions for both files. Put them in the root of the app directory, not in public: the Next.js docs say static metadata files like robots.txt should use the special metadata files inside app.

For robots.txt, app/robots.ts returns a Robots object. rules can be one object or an array, and userAgent accepts a string or an array, so you can write separate groups for AI crawlers:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: '/admin/' },
      { userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'], allow: '/' },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: 'https://www.example.com/sitemap.xml',
  }
}

Next.js docs describe robots.js and sitemap.js as special Route Handlers that are cached by default unless they use a request-time API or dynamic config. For the sitemap, app/sitemap.ts returns an array of entries with url, lastModified, changeFrequency, priority, and optional alternates, images and videos. For large sites, nest sitemap files in route segments or use generateSitemaps; the docs' example splits at Google's 50,000 URL limit and serves files at paths like /product/sitemap/1.xml.

Check what each AI crawler token controls before you block it. In the example above, blocking GPTBot opts out of OpenAI training while OAI-SearchBot stays allowed for ChatGPT search.

Serving llms.txt from Next.js

There is no llms.txt file convention in Next.js. The llms.txt proposal asks for a Markdown file at the site root, and you have two ways to serve it:

• Static file. Files in public are served from the base URL, so public/llms.txt becomes /llms.txt. Next.js applies Cache-Control: public, max-age=0 to public files.
• Route handler. Create app/llms.txt/route.ts and return a Response with a text/plain content type. Next.js docs show the same pattern for app/rss.xml/route.ts. Since Next.js 15, GET route handlers are dynamic by default, so add export const dynamic = 'force-static' if the file only changes at build time.

A route handler lets you build the list from the same content source as your sitemap, so the two never drift apart. Test the result with the llms.txt checker.

JSON-LD in Next.js

Next.js outputs no structured data on its own. The official guide recommends rendering JSON-LD as a native <script type="application/ld+json"> tag inside layout.js or page.js, not with next/script, because JSON-LD is data rather than executable code. The guide warns that JSON.stringify does not sanitize strings, and its example replaces < with <. For typing, it points to the community schema-dts package. Put Organization data in the root layout and page-level types (Product, Article, FAQPage) in each page.

Rendering and AI crawlers

This is where Next.js has an advantage. Layouts and pages are Server Components by default, and Client Components are still prerendered to HTML on the server. Vercel's December 2024 study of AI crawler traffic reported that none of the major AI crawlers it tested render JavaScript: ChatGPT's and Claude's crawlers fetch JavaScript files but do not execute them. Google's Gemini was the exception because it uses Googlebot's infrastructure.

The common mistake is fetching key content in the browser, for example product details or pricing loaded in useEffect. That text is missing from the HTML an AI crawler receives. Keep anything you want quoted in AI answers in Server Components or in the initial render. See JavaScript SEO for the wider background, and run the robots.txt checker after deploying to confirm the live file says what robots.ts intended.