topic Re: robots.txt and sitemap.xml are not updating even after modified in Settings in Blog, Website & Page Publishing

robots.txt and sitemap.xml are not updating even after modified in Settings

TFin — Wed, 02 Apr 2025 09:42:10 GMT

I have modified the robots.txt file [from "Disallow: /" to "Allow: /" ] as per attached screenshot but https://hubspotpagebuilder.com is not updating.

Also I have modified the sitemap.xml file, but it is showing "404 Not Found".

An early solution is highly appreciated since the Google Search Console is showing error of blocking of googlebot by hubspotpagebuilder.com .

Re: robots.txt and sitemap.xml are not updating even after modified in Settings

Jaycee_Lewis — Wed, 02 Apr 2025 22:03:15 GMT

Hi, @TFin 👋 Welcome to our community. Thanks for your question! I'd like to invite some of our community champions to the conversation — hey @Phil_Vallender @franksteiner79 do you have any suggestions for @TFin? Thank you for taking a look! — Jaycee

Re: robots.txt and sitemap.xml are not updating even after modified in Settings

TFin — Fri, 04 Apr 2025 06:47:41 GMT

The attached two screenshots says that even after passing of more than 24 hours, the settings have not been updated wherein robots.txt is modified from "Disallowed: /" to "Allowed: /" whereas the settings page provides the options of updating the sitemap.xml as "Always", but not updating live. When we update content of our website, the same is updated LIVE or INSTANTLY, then why the SITEMAP.XML and ROBOTS.TXT is not updating INSTANTLY ?

Pl ask your technical team to make changes in Settings options so that modification in the settings page updated instantly as per the options provided in therein.

Thanks

Re: robots.txt and sitemap.xml are not updating even after modified in Settings

TFin — Tue, 08 Apr 2025 10:09:24 GMT

Hi, @Jaycee_Lewis,

I have contacted Hubspot Customer Service and when they unable to resolve, I have posted my query related to blocking of googlebot by your hubspotpagebuilder.com on 2nd Apr'25 but even after passing of more than 6 days both Hubspot Customer Service & Community failed to resolve my query, which is not expected from such a large company.

A large company should have build their website settings in such a way that it would instantly modified on pressing SAVE button appearing in SETTINGS pages of ROBOTS.TXT & SITEMAP.XML without putting queries on Community page or contacting Customer Service.

And if Hubspot unable to update settings pages instantly / lively then they should not offer their customers to build a website on their portal.

Re: robots.txt and sitemap.xml are not updating even after modified in Settings

SanjayKumar — Thu, 17 Jul 2025 07:27:48 GMT

Hi @TFin,

HubSpot may cache the robots.txt and sitemap.xml files, so any updates can take up to 12–24 hours to appear. To check if your changes have been applied, try viewing the file in an incognito window or using a tool like curl

Make sure that the primary domain is connected properly. If your domain is using a redirect or is routed through a proxy service like Cloudflare, it may cause delays in reflecting file updates.

If your domain is behind Cloudflare or a content delivery network (CDN), ensure that you purge the cache to reflect any changes.

Sitemap Not Found (404 Fix)HubSpot auto-generates the sitemap only when: You have live published pages on your domain and your domain is set as primary

Make sure:At least one page is published and Sitemap generation is enabled in.