Vercel SEO for AI crawlers: previews, noindex and bot protection

Vercel serves what your framework builds

Vercel does not write robots.txt, sitemaps or llms.txt for you. They come from the framework you deploy: app/robots.ts and app/sitemap.ts in Next.js, the Nuxt SEO modules in Nuxt, or @astrojs/sitemap in Astro. For AI search the more important framework choice is rendering. Vercel's own December 2024 study of crawler traffic found that none of the major AI crawlers it tested render JavaScript, with Google's Gemini as the exception. Server-rendered or prerendered pages reach them; client-only pages do not.

What Vercel adds is a set of platform behaviors that change what crawlers receive.

Preview deployments and the noindex header

Vercel adds X-Robots-Tag: noindex to every deployment from a non-production branch, and to the previous production deployment once you promote a newer one. That keeps preview URLs out of search indexes without any work.

Two cases need attention:

• Custom domain on a non-production branch. Vercel omits the header when a custom domain such as staging.example.com is assigned to a non-production branch, so search engines can index it. Add the header yourself, for example in your framework's headers config gated on the environment.
• The vercel.app production URL. Vercel's duplicate content guide does not describe an automatic noindex for it. It says duplicates do not automatically incur a penalty but can split signals, and recommends a permanent, hostname-scoped redirect to your custom domain.

A noindex header is not access control. Anyone with the URL can still open the page, which is what Deployment Protection is for.

Deployment Protection and crawlers

Deployment Protection requires authentication for every request to a protected URL. Standard Protection, the option Vercel recommends for most projects, covers all deployments except production domains. All Deployments protects production domains too. With All Deployments on, crawlers cannot pass that authentication step, so AI engines cannot read the site at all. Use it for internal projects, not public marketing sites.

Firewall rulesets that affect AI crawlers

Vercel's firewall has two managed rulesets, both available on all plans and both inactive by default:

Ruleset	Default	What it does to crawlers
Bot Protection	Off	Challenges non-browser traffic; automatically excludes verified bots
AI bots	Allow	Log or deny known AI bots used for training, search and user fetches

Vercel's verified bots directory includes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot, so turning on Bot Protection should not challenge them. Attack Mode also lets known bots through. The AI bots ruleset is the one to handle with care: deny mode covers search and user-initiated fetches as well as training, so it can remove you from ChatGPT, Claude and Perplexity answers. If you only want to opt out of training, a robots.txt rule for the training token, or a custom WAF rule matching that user agent, is narrower.

Vercel also notes that Bot Protection does not work well behind a reverse proxy such as Cloudflare. If you run Cloudflare in front of Vercel, its AI bot settings apply first. After any change, run the robots.txt checker against your production domain.