topic Re: Robots txt is set to Disallow in Content Strategy & SEO

Robots txt is set to Disallow

AVision — Sun, 02 Feb 2025 17:34:58 GMT

My blog is recently unable to read by google

https://adventure-vision-treks-and-travels-46145676.hubspotpagebuilder.com/

the default value in robots.txt is as follow

User-agent: *
Disallow: /

while I set as following in Setting > Content > Pages >SEO & Crawelers

its not being reflected on live and my website is unable to be indexed on google.

is the are changes on hubspot free pages policy?

Re: Robots txt is set to Disallow

DianaGomez — Mon, 03 Feb 2025 13:10:40 GMT

Hi @AVision,

Thank you for reaching out to the Community!

I wanted to tag in a couple of subject matter experts to see if they have any input on this matter:

Hi @alyssamwilie , @Lucila-Andimol, @BarryGrennan, @JuanGomez0818 do you have any advice for @AVision? Thank you!

Cheers
Diana, Community Team

Re: Robots txt is set to Disallow

FarioConsulting — Mon, 03 Feb 2025 13:46:15 GMT

Hi,

You will want to ensure that your settings override the default ones for a specific domain....you would need to click "Override default settings" after selecting the domain whose robots.txt file you wish to edit

Cheers,
Harsh
Fario Consulting

Re: Robots txt is set to Disallow

Maxim1 — Wed, 26 Mar 2025 10:41:02 GMT

I have a similar problem on the site https://alterpex-48328568.hubspotpagebuilder.com/.

The problem appeared recently; previously, the site was indexed.

Account: free.

I contacted support, they advised me to ask a question in the community

Re: Robots txt is set to Disallow

AVision — Wed, 18 Jun 2025 06:44:04 GMT

Where can I change this setting?? Override default settings?? I can not find it on dashbord, can you guide me the steps

Re: Robots txt is set to Disallow

AVision — Wed, 18 Jun 2025 09:08:53 GMT

on setting > Content > Page > Domain
there this feature is disabled,

Re: Robots txt is set to Disallow

kennedyp — Wed, 18 Jun 2025 17:39:02 GMT

Thanks for the help so far, @FarioConsulting! Any insight regarding the updated information from @AVision? 😊

Re: Robots txt is set to Disallow

KTaksi — Fri, 28 Nov 2025 16:26:42 GMT

Looks like no solution on this topic 😒

Re: Robots txt is set to Disallow

BérangèreL — Mon, 01 Dec 2025 15:01:12 GMT

Hi @KTaksi and welcome, it's a pleasure to have you here!

Thanks for reaching out to the HubSpot Community!

Can you please share the steps you have already taken to have the robots txt set to disallow?

The more info, screenshots (without sensitive/confidential information), and details you can provide, the better the Community can assist.

Here are some resources for reference:

- Content Search
- Website Settings

Let's invite our Top Experts to this discussion: Hi @RidhiSharma2000, @Christensen and @Anton do you have suggestions to help @KTaksi, please?

Thanks so much for your help and have a fabulous day! 🦌
Bérangère

Re: Robots txt is set to Disallow

Anton — Mon, 01 Dec 2025 15:10:23 GMT

Hey @KTaksi,

as far as I remeber, hubspotpagebuilder.com domains are excluded from crawling.

In order to get your page crawled, you have to connect your domain to HubSpot.

Best, Anton

Re: Robots txt is set to Disallow

Christensen — Mon, 01 Dec 2025 15:36:03 GMT

@AVision this are not "your" domain this are HubSpot System Domain which are meant to be staging domains. You can't also override the the default settings in Content > Pages > SEO & Crawlers for the System Domain. If it's crawlable previously then HubSpot must have rolled out an update and this means that you'll need to connect your own domain to fully use the HubSpot for your blogs and be visible to crawlers. If you are happy without search engine presence then you can continue to use the System Domain as is else connecting your own domain will be your only option.