GetPicked

AI search glossary

Crawlability and indexability: how pages get into search and AI

Crawlability is whether search engine and AI crawlers can discover and fetch a page. Indexability is whether a fetched page can be added to a search index and shown in results. A page must pass both before it can rank or be cited.

Also called: crawlability, indexability, crawlable, indexable, crawl and index

Why it matters for AI search

Google says a page must be indexed and snippet-eligible to appear as a supporting link in AI Overviews or AI Mode, and AI engines that browse the web need to fetch a page before they can cite it. Crawlability and indexability are the floor of AI visibility: no amount of good content helps a page that bots cannot reach or that never makes it into the index.

How crawling and indexing work

Google describes Search in three stages:

  1. Crawling. Crawlers such as Googlebot discover URLs, mostly from links and sitemaps, and download the page's text, images and videos.
  2. Indexing. Google analyzes the content, groups duplicate pages, picks a canonical and stores the result in its index.
  3. Serving. When someone searches, Google returns the indexed pages it judges most relevant.

Crawlability covers the first stage, indexability the second. Google is clear that neither is guaranteed: it says indexing is not guaranteed and that not every page it processes will be indexed.

Common crawl blockers

  • robots.txt rules that disallow the URL or the whole directory.
  • Server errors. Google says 500-level errors tell its crawlers to slow down.
  • Login walls and pages only reachable through forms or search boxes.
  • Orphan pages with no internal links pointing to them and no sitemap entry.
  • Content that needs JavaScript. Google renders JavaScript, but without rendering it may miss content loaded by scripts. See JavaScript SEO.

Common indexing blockers

  • noindex in a robots meta tag or X-Robots-Tag header.
  • Duplicates. Google keeps one version from each cluster as the canonical and treats the rest as alternates. A canonical tag states your preference.
  • Low-quality or thin content that Google decides not to store.
  • Redirects and errors, such as soft 404s, 404s and 5xx responses.

One trap catches many sites: if robots.txt blocks a URL, Googlebot never sees its noindex rule, so the URL can still appear in results when other sites link to it.

Google's AI features documentation sets the bar plainly: to be a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet, and there are no additional technical requirements. Other AI engines run their own crawlers and fetchers, each with its own user agent, and they need to reach a page before they can quote it. Not all of them render JavaScript the way Google does, so server-rendered text is the safest choice.

How to check

  1. Open the Page indexing report in Search Console and read the reasons listed for non-indexed pages, such as "Discovered, currently not indexed" or "Duplicate without user-selected canonical".
  2. Run important URLs through the URL Inspection tool to see what Googlebot fetched and rendered.
  3. Keep an accurate XML sitemap and link to key pages from your navigation.
  4. Test access for AI bots with the AI crawler checker.

For faster discovery of changed pages on engines that support it, see IndexNow. Once a page is crawled and indexed, whether it gets used in AI answers depends on LLM grounding and how well it answers the question.

Questions

What is the difference between crawlability and indexability?
Crawlability is about access: can a bot find and download the page. Indexability is about eligibility: once fetched, can the page be stored and shown. A page can be crawlable but not indexable, for example when it carries a noindex rule or is treated as a duplicate.
Should every page on my site be indexed?
No. Google's Page indexing report help says not to expect every URL to be indexed. Duplicates, filtered views and thin utility pages are often left out on purpose. What matters is that your important pages are indexed.
What does Crawled, currently not indexed mean?
Google fetched the page but chose not to index it for now. It may be indexed later. Improving the page's usefulness and its internal links is usually a better fix than resubmitting it.
Does JavaScript hurt crawlability?
It can. Google renders pages with a recent version of Chrome, but content that only appears after scripts run can be missed, and some AI crawlers may not run JavaScript at all. Server-render the content that matters.

Related terms

Sources