Googlebot: Google's search crawler, and what blocking it costs

Googlebot and Google's other tokens

Google runs many crawlers, but two tokens matter most when you write rules for search and AI:

Token	What it controls	Block it if…
Googlebot	Crawling for Google Search (all Search features, including AI Overviews and AI Mode), Discover, Images, Video mode, News	You want pages out of Google entirely (rarely what a business wants)
Google-Extended	Whether crawled content may train future Gemini models and ground answers in Gemini Apps and Vertex AI	You don't want Gemini training or Gemini grounding to use your content
Googlebot-Image	Image crawling; Google lists Googlebot as its fallback token	You want images out of Google Images but pages left in

Google-Extended sends no requests of its own. Every fetch you see in your logs comes from Googlebot or another existing Google user agent, and the token only tells Google how it may use what was crawled.

What blocking Googlebot does, and what it doesn't

A Disallow for Googlebot stops Google crawling the matched URLs for Search and everything Google lists under it: Discover, Images, Video mode and News. Because AI Overviews and AI Mode are part of Google Search, they rely on what Googlebot crawls, and Google-Extended does not cover them. Blocking Googlebot to avoid AI Overviews therefore also costs you the ordinary blue links.

Blocking Googlebot is the wrong tool for a training opt-out. Gemini training and grounding are governed by the Google-Extended token, so to stay in Search but out of Gemini training, allow Googlebot and disallow Google-Extended.

User agent variants

Google documents two main strings. The smartphone crawler is the one in the frontmatter above. The desktop crawler sends:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Rarely you will also see Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or the bare Googlebot/2.1 (+http://www.google.com/bot.html). W.X.Y.Z is a placeholder for the Chrome version, so log filters and firewall rules should match Googlebot/2.1 or use a wildcard, not an exact version.

How to block or allow Googlebot

robots.txt rules go at the root of each host. To keep Googlebot out of one directory while allowing the rest:

User-agent: Googlebot
Disallow: /internal/

A group that names Googlebot replaces your User-agent: * rules for it, so repeat any shared Disallow lines inside the Googlebot group. Other crawlers read your Googlebot group too: Applebot follows Googlebot's rules when robots.txt doesn't name Applebot, so a Googlebot block can spill over.

Verifying a real Googlebot visit

Google says its common crawlers generally come from the IP ranges in common-crawlers.json, and their reverse DNS hostnames match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Do a reverse lookup on the request IP, check the domain, then a forward lookup on that hostname to confirm it resolves back to the same IP. The verification guide walks through both methods.

The docs have moved from Search Central to developers.google.com/crawling, and the old googlebot.json path now redirects to common-crawlers.json, so update any scripts that fetch the old URL. To see how your current robots.txt treats Googlebot, run it through the robots.txt checker.