Cloudflare robots.txt and AI bot blocking: check before AI search

Two layers: what Cloudflare does before robots.txt

robots.txt is a request. Cloudflare's bot settings are enforcement: a blocked crawler gets an error before it reaches your page, whatever your robots.txt says. That is the risk for AI search. A site can have a perfect robots.txt that allows OAI-SearchBot and PerplexityBot, and still be invisible because a CDN setting chosen earlier now blocks them. Nothing in your CMS shows it.

Cloudflare's AI bot controls changed twice:

• July 1, 2025. Cloudflare announced it would block AI crawlers accessing content without permission or compensation by default, and that every new domain would be asked at sign-up whether to allow AI crawlers.
• September 15, 2026. The single Block AI bots setting was deprecated in favor of separate controls under Security Settings > Configure AI bot policies: Search (crawlers that index content to answer questions later), Agent (real-time activity on a person's behalf, such as chat fetch bots) and Training (crawlers taking content to train models, including mixed-purpose crawlers). Each can be set to Block, Block on pages with ads, or Allow.

Cloudflare's September 2026 post adds an important detail: Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot. If you block Training to stop AI training, you may also be blocking search. Disallow AI Training is the option that publishes a no-training preference in robots.txt while leaving search crawling alone.

Managed robots.txt: when Cloudflare edits your file

Managed robots.txt is available on all plans. If your origin returns a robots.txt, Cloudflare prepends its managed block; if not, Cloudflare creates one. Per the docs, the block includes Content-signal: search=yes, ai-train=no, use=reference under User-Agent: *, then Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.

Most of these are training crawlers or training opt-out tokens, so blocking them should not remove you from ChatGPT search or Claude search, which use separate tokens. But two things catch people out. Your own rules now appear below Cloudflare's block, so a User-agent: * group you wrote may not read the way you expect. And anyone editing robots.txt at the origin will not see the prepended lines. Fetch the live file through Cloudflare, or use the robots.txt checker, to see what crawlers actually receive.

AI Crawl Control and pay per crawl

AI Crawl Control (formerly AI Audit) is available on all plans. It shows which AI services visit your site and how often, lets you set allow or block rules per crawler, and reports which crawlers follow your robots.txt. Pay per crawl, part of the same product and in closed beta, lets you charge AI crawlers per zone: a crawler that sends payment intent gets HTTP 200, others get 402 Payment Required with the price. A WAF or bot management block overrides pay per crawl.

A quick AI search check for Cloudflare sites

1. Open Configure AI bot policies and confirm Search is set to Allow. Decide on Agent deliberately: Cloudflare describes Agent as chat fetch bots and browser-use agents, the kind of fetcher ChatGPT-User is when it opens a page a user asked about.
2. Use Disallow AI Training, not Block, if your goal is only to opt out of training.
3. Read your live robots.txt and look for the managed block at the top.
4. Review per-crawler rules in AI Crawl Control, and any custom WAF rules that match bot user agents.
5. Compare tokens against the AI crawler directory to see which ones feed search and which feed training.

Cloudflare does not generate sitemaps, llms.txt or structured data; those come from your site. If you deploy on Vercel behind Cloudflare, note Vercel's own bot controls as well.