Robots.txt issue

[rickbyrd]

The robots.txt file within Hubspot is not matching the robots.txt file in Google Search Console.  See images below.

Hubspot robots.txtGSC robots.txt

I do not know how lines 7 - 10 showed up in the GSC version.  I do not know how to remove them as they are not showing up in Hubspot.

Any thoughts?

Rick

[Phil_Vallender · Accepted solution]

Hi @rickbyrd

Those look like they are inserted by HubSpot automatically to prevent your post and page previews, email preferences pages and some other things from showing up in search. This is probably a good thing. Do you have a specific reason for wanting to remove them? 

[rickbyrd]

@Phil_Vallender,

They are showing a blocked resources in GSC so I wanted to get them cleared but there is no real good reason if these resources are blocked.  It is mostly the Recent Posts listings that are blocked.

Thanks for the explanation!

Rick

[mcoley]

@rickbyrd We block some of these resources because the URL's are signed in order to prevent an attacker from leveraging the service to DDOS a different website. GC would try to follow old expired links and throw errors, so we decided to just block them entirely since they are not important to the indexing of the page.