Why is Applebot obeying my Googlebot rules?

Apple says that if robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. Add a User-agent: Applebot group if you want it treated differently.

Does Applebot support Crawl-delay?

No. Apple states that Applebot does not follow crawl-delay. It says its crawl rate adjusts automatically when a site slows down or returns errors.

How do I keep my content out of Siri's AI answers but stay in search?

Use the nosnippet robots meta tag (or an X-Robots-Tag: applebot: nosnippet header) on that content. Apple says it won't use nosnippet data as context for AI-generated output, while the page stays discoverable.

What is the iTMS user agent coming from applebot.apple.com?

Apple uses iTMS to crawl URLs tied to registered Apple Podcasts content. Apple says it does not follow robots.txt because it is not a general search crawler.

Should I block JavaScript and CSS from Applebot?

No. Apple says Applebot may render pages in a browser, and blocking scripts, styles or XHR resources in robots.txt can stop it rendering your content properly.

Apple · AI search

Applebot: Apple's crawler for Siri and Spotlight, and its AI role

Applebot is Apple's web crawler: its data powers search in Spotlight, Siri and Safari, and may also train Apple's foundation models and give context to AI-generated answers. It respects robots.txt, and if your file doesn't name Applebot but does name Googlebot, it follows the Googlebot rules. To opt out of training only, disallow Applebot-Extended; to keep content out of AI answers, use nosnippet.

robots.txt token: Applebot
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Operator: Apple
Type: AI search
What it feeds: Search across Apple's ecosystem (Spotlight, Siri, Safari), and possibly training of Apple foundation models and context for AI-generated answers in Apple products.
Follows robots.txt: Yes. Apple says Applebot respects standard robots.txt directives targeted at it, and falls back to Googlebot's rules when Applebot isn't named. It does not follow crawl-delay.
How to verify it: Reverse DNS to a *.applebot.apple.com hostname plus a forward lookup, or match the IP against Apple's published CIDR list (list)

Should you block Applebot?

Blocking Applebot keeps your content out of Apple's search features (Spotlight, Siri, Safari suggestions), and with it out of Apple's AI training and AI answers, since all of them use Applebot's crawl. Most sites that only object to training should leave Applebot allowed and disallow Applebot-Extended instead.

Block Applebot

User-agent: Applebot
Disallow: /

Allow Applebot

User-agent: Applebot
Allow: /

Add these lines to /robots.txt at your site root. A group naming Applebot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.

Live check

Is Applebot allowed on your site?

Applebot, Applebot-Extended and nosnippet

Apple has one crawler and two separate controls over how its crawl is used:

Control	What it governs	Use it if…
Applebot (robots.txt)	Whether Apple crawls at all, which feeds Spotlight, Siri, Safari, AI training and AI answers	You want Apple out completely
Applebot-Extended (robots.txt)	Whether Applebot's data may train Apple's general-purpose foundation models	You object to training but want Apple search visibility
nosnippet (meta tag or header)	Whether content is used as context for AI-generated answers, and whether a description or web answer is shown	You want to stay findable but out of AI answers

Apple notes that even with Applebot-Extended disallowed and nosnippet applied, Applebot can still crawl, and your content remains discoverable through Spotlight, Siri and Safari.

What blocking Applebot does, and what it doesn't

Apple says enabling Applebot "allows website content to appear in search results for Apple users." It doesn't spell out the reverse, but the consequence of blocking follows: no Applebot crawl means no Apple search presence, and no input to Apple's AI training or AI answers either, since those all use Applebot-crawled data. A block does not stop the iTMS agent for Apple Podcasts, which Apple says ignores robots.txt.

How to block or allow Applebot

Apple's example gives Applebot its own group:

User-agent: Applebot
Allow: /
Disallow: /private/

User-agent: *
Disallow: /not-allowed/

Two quirks to know. First, the Googlebot fallback: with no Applebot group, Applebot reads your Googlebot group, so a Disallow you wrote for Google also applies to Apple. Second, Applebot ignores Crawl-delay, so you can't throttle it that way. Apple also asks you not to block the JavaScript, CSS and XHR resources your pages need, because Applebot may render pages.

For page-level control, Applebot supports noindex, nosnippet, nofollow, none and all in robots meta tags (including a meta name="applebot" form) and in the X-Robots-Tag header, for example X-Robots-Tag: applebot: nosnippet on a PDF. Pages marked isAccessibleForFree: false in schema.org markup can appear in results, but Apple won't use them as context for AI answers.

User agent strings

Apple's desktop example is in the frontmatter. The mobile example is:

Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Apple says the advertised browser version changes from time to time within the same format, so match on Applebot, not the whole string.

Verifying a real Applebot visit

Apple says Applebot traffic is generally identified by reverse DNS in the applebot.apple.com domain. Run host on the IP, check the hostname, then run host on that hostname and confirm it returns the same IP. Alternatively, match the IP against Apple's CIDR list. To check how your robots.txt (including any Googlebot fallback) treats Applebot, use the robots.txt checker.

Questions

Why is Applebot obeying my Googlebot rules?: Apple says that if robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. Add a User-agent: Applebot group if you want it treated differently.
Does Applebot support Crawl-delay?: No. Apple states that Applebot does not follow crawl-delay. It says its crawl rate adjusts automatically when a site slows down or returns errors.
How do I keep my content out of Siri's AI answers but stay in search?: Use the nosnippet robots meta tag (or an X-Robots-Tag: applebot: nosnippet header) on that content. Apple says it won't use nosnippet data as context for AI-generated output, while the page stays discoverable.
What is the iTMS user agent coming from applebot.apple.com?: Apple uses iTMS to crawl URLs tied to registered Apple Podcasts content. Apple says it does not follow robots.txt because it is not a general search crawler.
Should I block JavaScript and CSS from Applebot?: No. Apple says Applebot may render pages in a browser, and blocking scripts, styles or XHR resources in robots.txt can stop it rendering your content properly.

Related crawlers

Sources

Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.