GetPicked

AI search glossary

Robots meta tag: noindex, nosnippet, X-Robots-Tag and AI features

The robots meta tag is an HTML meta element (name="robots") in a page's head that tells search engine crawlers whether to index the page and how it may be shown, for example noindex, nofollow, nosnippet or max-snippet. The X-Robots-Tag HTTP header carries the same rules.

Also called: meta robots, noindex tag, X-Robots-Tag, nosnippet, robots directives

Why it matters for AI search

These rules decide whether Google's AI features can use your page at all. Google says nosnippet applies to AI Overviews and AI Mode and stops the content being used as a direct input, and max-snippet caps how much they can use. A page also needs to be indexed and snippet-eligible to be a supporting link. One stray noindex or nosnippet in a template can remove a whole section of a site from AI answers.

How the robots meta tag works

The tag sits in the page's <head>:

<meta name="robots" content="noindex, nofollow">

Using name="robots" addresses all crawlers that support the rule. Using name="googlebot" addresses only Google. Rules can be comma-separated in one tag or split across several, and when they conflict Google applies the more restrictive one. Crawlers can only read the tag if they can fetch the page, which is why a robots.txt block hides it.

The same rules can be sent as an HTTP response header, which is the only option for non-HTML files:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

The rules that matter most

  • noindex: keep the page out of search results. none is shorthand for noindex, nofollow.
  • nofollow: do not follow the links on the page.
  • nosnippet: show no text snippet or video preview. A static image thumbnail may still appear.
  • max-snippet:[number]: cap the snippet at that many characters. 0 behaves like nosnippet and -1 lets Google choose.
  • max-image-preview: none, standard or large.
  • unavailable_after: drop the page from results after a date and time.
  • data-nosnippet: an HTML attribute, not a meta rule. Put it on a span, div or section to keep just that text out of snippets while the rest of the page stays eligible.
<p>Our plan costs $49 a month. <span data-nosnippet>Internal promo code details.</span></p>

Google says it ignores the old noarchive and nocache rules in Search.

What it means for Google AI features

Google's AI features documentation lists only two requirements for a page to appear as a supporting link in AI Overviews or AI Mode: it must be indexed, and it must be eligible to show with a snippet. That puts the robots meta tag in direct control:

  • noindex removes the page from Search, so it cannot be a supporting link.
  • nosnippet applies across Google surfaces including AI Overviews and AI Mode, and stops the content being used as a direct input for them.
  • max-snippet also applies to those features and limits how much of the content they can use as direct input, unless you have granted permission another way, such as a license agreement.
  • data-nosnippet lets you hold back specific passages without losing the whole page.

Robots rules generally do not restrict structured data used for rich results, except description-style values that max-snippet can limit.

How to check and fix

  1. Use the URL Inspection tool in Search Console to see the HTML Googlebot received and confirm which rules it found.
  2. Check the Page indexing report for URLs flagged with noindex. Look for template-wide mistakes, such as a staging noindex that shipped to production.
  3. Check response headers too: an X-Robots-Tag set at the CDN will not show in the HTML.
  4. Do not block a URL in robots.txt if you need its noindex to be seen.
  5. After a change, allow time for recrawling. Google says it can take several days to several months for AI features to reflect new preview controls.

Robots rules are one half of crawlability and indexability. For choosing between duplicate URLs, use the canonical tag rather than noindex. To check which AI bots can reach your pages, try the AI crawler checker.

Questions

What is the difference between noindex and robots.txt?
Robots.txt controls crawling, noindex controls indexing. Google does not support noindex inside robots.txt, and if robots.txt blocks a URL, Googlebot never sees the page's noindex rule, so the URL can still appear in results when other sites link to it.
Does nosnippet remove my page from AI Overviews?
Google says nosnippet applies to AI Overviews and AI Mode and prevents the content from being used as a direct input for them. Since a supporting link must be eligible for a snippet, nosnippet effectively takes the page out of those features.
When should I use X-Robots-Tag instead of the meta tag?
Use the header for files that have no HTML head, such as PDFs, images and videos, or when it is easier to set rules at the server or CDN level for many URLs at once.
What happens if two robots rules conflict?
Google applies the more restrictive rule. For example, if one tag allows snippets and another sets nosnippet, the page gets no snippet.

Related terms

Sources