AI search readiness · Hosting and CDN
GitHub Pages SEO: robots.txt, jekyll-sitemap and llms.txt
GitHub Pages is static hosting: it serves the HTML, CSS and JavaScript files in your repository or build output, so content is already in the HTML that AI crawlers read. You add robots.txt and llms.txt as plain files. Jekyll sites get a sitemap and JSON-LD from the supported jekyll-sitemap and jekyll-seo-tag plugins.
- Edit robots.txt
- YesCommit robots.txt to the root of the publishing source. jekyll-sitemap also generates a basic one if none exists. It only counts at the root of a host.
- XML sitemap
- With a plugin or appAdd jekyll-sitemap to plugins in _config.yml and set url. Other generators (via GitHub Actions) need their own sitemap tool.
/sitemap.xml - llms.txt
- YesCommit a static llms.txt to the root of the publishing source.
- AI crawlers
- No AI-specific setting documented. GitHub Pages serves the same static files to every client; any AI rules go in your robots.txt.
- Structured data
- None by default. jekyll-seo-tag outputs JSON-LD site and post metadata when you add the {% seo %} tag before </head>; otherwise add a script tag yourself.
How GitHub Pages serves your site
GitHub describes Pages as a static site hosting service that takes HTML, CSS and JavaScript files straight from a repository. You can publish the files as they are, let GitHub build a Jekyll site, or run any generator in a GitHub Actions workflow, which GitHub's docs now describe as the recommended way to deploy and automate Pages sites.
Static hosting suits AI search. Each URL returns a finished HTML file, so crawlers that do not execute JavaScript, such as GPTBot and ClaudeBot, see the same text a visitor does. The exception is a client-rendered single-page app deployed to Pages; see the React guide for why that is a problem and how to prerender.
robots.txt on GitHub Pages, and the project-site trap
Commit a robots.txt file to the root of your publishing source (the repository root, the /docs folder, or your build output). If you use jekyll-sitemap and have no robots.txt, the plugin's source shows it generates one, but only when the site does not already have a robots.txt file.
Where the file ends up matters. Google's robots.txt documentation says the file must be at the root of the host it applies to and cannot be placed in a subdirectory. GitHub Pages has three shapes:
| Site type | URL | Does your robots.txt apply? |
|---|---|---|
| User or organization site | username.github.io | Yes, it is at the host root |
| Project site | username.github.io/repo/ | No, it sits at /repo/robots.txt |
| Any site with a custom domain | www.example.com | Yes |
For a project site without a custom domain, crawler rules belong in the robots.txt of your username.github.io repository. Test the live result with the robots.txt checker.
Sitemap with jekyll-sitemap
GitHub Pages enables a fixed set of Jekyll plugins by default and lets you add others from its supported list. jekyll-sitemap is on that list (version 1.4.0 on the dependency versions page). To enable it:
- Add
jekyll-sitemapunderpluginsin_config.yml. The plugin's README warns that the GitHub Pages gem ignores plugins listed only in the Gemfile. - Set
url: "https://www.example.com"in_config.ymlso links are absolute. - Build. The plugin writes
sitemap.xmlto the site root.
Exclude a page with sitemap: false in its front matter. If you build with Hugo, Eleventy or another generator through Actions, use that tool's own sitemap feature instead. Either way, add a Sitemap: line to robots.txt. See XML sitemap for what to include.
llms.txt and structured data
GitHub Pages has no special handling for llms.txt. Commit a Markdown file named llms.txt to the root of your publishing source and it is served at /llms.txt.
For structured data, jekyll-seo-tag (also on the supported list, version 2.8.0) adds the page title, meta description, canonical URL, Open Graph and Twitter tags, and JSON-LD site and post metadata. Add it to plugins in _config.yml and place the {% seo %} tag right before the closing head tag in your layout, as its installation docs instruct. For other types, such as Product or FAQPage, paste a <script type="application/ld+json"> block into the relevant layout or page.
Check your GitHub Pages site
Paste your domain to see which AI crawlers your live robots.txt lets in.
robots.txt checker
Test your robots.txt against every AI crawler
robots.txt generator
# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training
User-agent: *
Allow: /
# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Questions
- Does GitHub Pages create a sitemap automatically?
- No. For Jekyll sites, enable jekyll-sitemap by adding it to plugins in _config.yml and setting url; it is on the GitHub Pages supported plugin list. Sites built with other generators through GitHub Actions need that generator's sitemap feature.
- Why is my robots.txt ignored on a GitHub Pages project site?
- A project site lives at username.github.io/repository/, so its robots.txt is at /repository/robots.txt. Google's docs say robots.txt must be at the root of the host and cannot sit in a subdirectory. Use a custom domain, or edit the robots.txt of your username.github.io site.
- Can AI crawlers read a GitHub Pages site?
- Generally yes, because GitHub Pages serves prebuilt static files. If your site is a JavaScript single-page app that builds content in the browser, crawlers that do not run JavaScript will see little text.
- Can I add llms.txt to GitHub Pages?
- Yes. Commit llms.txt to the root of your publishing source and it is served at /llms.txt like any other static file.
Other hosting and cdn
Related
Sources
Checked against these pages on 2026-10-19. Operators change their crawlers; check the source before relying on a detail.