X-Robots tag is being applied to blog post in some cases

[TMorris7]

Hello

We are currently working on a clients website and one of their blog articles has not ranked. After some investigation we have found that when the article was added to a campaign the 'X-Robots-Tag: none' was added to the article preventing it from being indexed.

We have since removed the article from the campaign and now get the following result:

Does anyone have any insight in to why this happened and what we can do to prevent it in the future?

We are going to attempt to add the article back in to the campaign and see if the 'X-Robots-Tag' is re-added. As this is a tag in the http response which is set in the htaccess or conf.d file there isn't a lot we can do to prevent it from a server side on a hosted site.

Any information would be appreciated.

Thank you

Trudi

[TMorris7]

Hi @MiaSrebrnjak 

The X-Robots-Tag comes from the httpd or conf.d files which are set at a server level which users of HubSpot do not have access to.

We are not trying to block the page from being seen by search engines we are trying to make it visible and even having a robots.txt which allows crawling of all pages didn't help. It is one of our clients core articles which we want to rank highly.

It was only after running a Lighthouse report we found that the X-Robot-Tag had been set at a server level (this site is hosted entirely on HubSpot). This has been causing us major headaches to try and fix for our client. We have found that removing the article from the campaign it was created under and then re-adding it to the campaign removes the X-Robots tag. I believe this is a bug on HubSpots side because it is independant of the robots.txt file. Should I raise this in a seperate forum?

Thank you

Trudi

[MiaSrebrnjak]

Hi @TMorris7,

Thank you for the additional information. I would like to send this to our content tool team for further investigation - could you send me links to your client's pages in a direct message for more context?  

If possible, please include a link to the article where you were able to remove the X-Robots tag by removing and re-adding it to the campaign. 

Thank you!

Mia, Community Team 

[MiaSrebrnjak]

@TMorris7,

Thank you for reaching out to the Community!  

The the 'X-Robots-Tag' is one of the possible methods to prevent search engines from indexing specific pages on your website. You can read more about this topic in this Knowledge Base article. You'll also find instructions on how to set up the robots.txt file in HubSpot in this article. 

There are a number of occasions where you may want to exclude a webpage -- or a portion of a webpage -- from search engine crawling and indexing. I want to highlight @Esther_ISM's answer in this Community post, where she shared a couple of great reasons and use cases. 

I hope this helps!

Cheers
Mia, Community Team