Where do I edit robots.txt in HubSpot?

Click the settings icon, go to Content > Pages, pick a domain (or Default settings for all domains), open the SEO & Crawlers tab and edit the Robots.txt field. Click Save. To give one domain its own file, select it and click Override default settings first.

Why are my HubSpot landing pages missing from the sitemap?

HubSpot adds live website pages and blog posts automatically but not landing pages. Add them at Settings > Content > Domains & URLs > Sitemap with + Add a page, then click Save. Changes can take up to 30 minutes to appear.

Can I host llms.txt on HubSpot?

Not at the domain root. HubSpot's own documentation says files like llms.txt that must sit at the root have to be hosted externally, because uploaded files get generated /hubfs/ URLs. Some teams use a redirect or a Marketplace app, which HubSpot does not document as a supported method.

Does HubSpot add schema markup automatically?

Only for blogs. The Structured Data switch in blog settings adds BlogPosting and is on by default for blogs created after Aug. 4, 2022. Turn it off if you already output custom blog schema, since HubSpot warns that duplicates can cause problems.

AI search readiness · Website builders and CMSs

HubSpot CMS SEO: robots.txt, sitemap, llms.txt and blog schema

HubSpot's Content Hub lets you edit robots.txt per domain under Settings > Content > Pages > SEO & Crawlers, and builds /sitemap.xml from live website pages and blog posts (landing pages must be added by hand). HubSpot itself says a root-level file such as llms.txt must be hosted outside HubSpot. Blogs can switch on BlogPosting structured data; other schema goes in Head HTML.

Edit robots.txt: YesSettings > Content > Pages > choose a domain > SEO & Crawlers > Robots.txt. Default settings apply to all connected domains unless overridden.
XML sitemap: YesAutomatic for live website pages and blog posts; landing pages are added manually. Managed at Settings > Content > Domains & URLs > Sitemap. /sitemap.xml
llms.txt: NoHubSpot says plain-text files required at the domain root, such as llms.txt, must be hosted externally. Files uploaded to HubSpot get generated /hubfs/ URLs.
AI crawlers: No AI-specific setting documented. AI crawlers are managed like any bot, with User-agent groups in the robots.txt field. Keep HubSpotContentSearchBot allowed if you use HubSpot site search.
Structured data: Blogs have a Structured Data switch (Content > Blog, General tab) that adds BlogPosting; on by default for blogs created after Aug. 4, 2022. Other JSON-LD goes in a page's Head HTML or the domain's Site Header HTML.

HubSpot robots.txt: where it lives

HubSpot keeps robots.txt in settings, not in the file manager. It is available on all HubSpot products and plans:

Click the settings icon in the top navigation.
In the left sidebar go to Content > Pages.
In Choose a domain to edit its settings, keep Default settings for all domains or pick one domain.
Open the SEO & Crawlers tab and edit the Robots.txt field.
Click Save.

Default settings apply to every connected domain. Selecting one domain and clicking Override default settings gives that domain its own file, which replaces the default entirely, so copy across any shared rules. Only content on domains connected to HubSpot can be controlled here, and HubSpot system domains containing "hs-sites" are always noindexed.

HubSpot documents no AI crawler switch. To set a policy, add groups to the field like any other rule:

User-agent: GPTBot
Disallow: /

User-agent: HubSpotContentSearchBot
Allow: /

The second group matters if you use HubSpot's site search module, which relies on that bot. Before blocking anything, read what each bot does: GPTBot collects training data, while OAI-SearchBot decides whether ChatGPT search can cite you.

HubSpot also warns against combining robots.txt blocks with noindex. To drop a page that is already indexed, put <meta name="robots" content="noindex"> in its Head HTML (Settings > Advanced) and leave it crawlable so search engines can see the tag.

HubSpot sitemap

Each HubSpot-hosted domain has a sitemap at /sitemap.xml. Live website pages and blog posts are added automatically, along with HubDB dynamic pages whose parent is listed and video entries for videos on HubSpot pages. Landing pages are not, so a campaign page you want AI engines and search engines to find has to be added by hand. Manage it at Settings > Content > Domains & URLs > Sitemap: + Add a page to include one, Remove to drop one (which does not noindex it), and Options for change frequency and priority. Save, allow up to 30 minutes, then submit /sitemap.xml in Google Search Console. See XML sitemap for what to include.

llms.txt on HubSpot

HubSpot's file documentation is direct: a plain-text file that must sit at the domain root, such as llms.txt, has to be hosted outside HubSpot, because anything uploaded to the Files tool gets a generated /hubfs/ path. If you want an llms.txt, host it on another server or proxy the path, and confirm the live result with the llms.txt checker.

Structured data

Blogs have a Structured Data switch on the General tab of blog settings (Content > Blog) that adds BlogPosting markup. It is on by default for blogs created after Aug. 4, 2022; older blogs may need it turned on. Elsewhere, add JSON-LD yourself: per page in Settings > Advanced > Head HTML, or for a whole domain in Site Header HTML under Content > Pages. A homepage Organization schema block and FAQ markup on pages that genuinely answer questions are sensible starting points. Don't add your own BlogPosting while the switch is on.

Check your HubSpot CMS site

Paste your domain to see which AI crawlers your live robots.txt lets in.

robots.txt checker

Test your robots.txt against every AI crawler

robots.txt generator

Allow AI search, block trainingStay citable in ChatGPT, Claude, Perplexity, Gemini and Copilot answers while opting out of model training.Allow everythingEvery crawler may read every public page. The default when there is no robots.txt.Block all AI crawlersKeep Google and Bing search, block every AI crawler. AI answers will rarely cite or recommend the site.

Sitemap URLPrivate paths (one per line)

robots.txt

# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training

User-agent: *
Allow: /

# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Questions

Where do I edit robots.txt in HubSpot?: Click the settings icon, go to Content > Pages, pick a domain (or Default settings for all domains), open the SEO & Crawlers tab and edit the Robots.txt field. Click Save. To give one domain its own file, select it and click Override default settings first.
Why are my HubSpot landing pages missing from the sitemap?: HubSpot adds live website pages and blog posts automatically but not landing pages. Add them at Settings > Content > Domains & URLs > Sitemap with + Add a page, then click Save. Changes can take up to 30 minutes to appear.
Can I host llms.txt on HubSpot?: Not at the domain root. HubSpot's own documentation says files like llms.txt that must sit at the root have to be hosted externally, because uploaded files get generated /hubfs/ URLs. Some teams use a redirect or a Marketplace app, which HubSpot does not document as a supported method.
Does HubSpot add schema markup automatically?: Only for blogs. The Structured Data switch in blog settings adds BlogPosting and is on by default for blogs created after Aug. 4, 2022. Turn it off if you already output custom blog schema, since HubSpot warns that duplicates can cause problems.

Other website builders and cmss

Sources

Checked against these pages on 2026-10-19. Operators change their crawlers; check the source before relying on a detail.