X-Robots tag is being applied to blog post in some cases

TMorris7
Member

Hello

 

We are currently working on a clients website and one of their blog articles has not ranked. After some investigation we have found that when the article was added to a campaign the 'X-Robots-Tag: none' was added to the article preventing it from being indexed.

Screenshot 2021-10-07 at 14.23.49.png

 

We have since removed the article from the campaign and now get the following result:
Screenshot 2021-10-07 at 14.24.57.png

 

Does anyone have any insight in to why this happened and what we can do to prevent it in the future?

We are going to attempt to add the article back in to the campaign and see if the 'X-Robots-Tag' is re-added. As this is a tag in the http response which is set in the htaccess or conf.d file there isn't a lot we can do to prevent it from a server side on a hosted site.

Any information would be appreciated.

Thank you

Trudi

SEO
3 Replies 3
MiaSrebrnjak
Community Manager

@TMorris7,

 

Thank you for reaching out to the Community!  

The the 'X-Robots-Tag' is one of the possible methods to prevent search engines from indexing specific pages on your website. You can read more about this topic in this Knowledge Base article. You'll also find instructions on how to set up the robots.txt file in HubSpot in this article. 

 

There are a number of occasions where you may want to exclude a webpage -- or a portion of a webpage -- from search engine crawling and indexing. I want to highlight @Esther_ISM's answer in this Community post, where she shared a couple of great reasons and use cases. 

 

I hope this helps!

 

Cheers
Mia, Community Team

 

 


Wusstest du, dass es auch eine DACH-Community gibt?
Nimm an regionalen Unterhaltungen teil, indem du deine Spracheinstellungen änderst !


Did you know that the Community is available in other languages?
Join regional conversations by changing your language settings !


TMorris7
Member

Hi @MiaSrebrnjak 

The X-Robots-Tag comes from the httpd or conf.d files which are set at a server level which users of HubSpot do not have access to.

We are not trying to block the page from being seen by search engines we are trying to make it visible and even having a robots.txt which allows crawling of all pages didn't help. It is one of our clients core articles which we want to rank highly.

It was only after running a Lighthouse report we found that the X-Robot-Tag had been set at a server level (this site is hosted entirely on HubSpot). This has been causing us major headaches to try and fix for our client. We have found that removing the article from the campaign it was created under and then re-adding it to the campaign removes the X-Robots tag. I believe this is a bug on HubSpots side because it is independant of the robots.txt file. Should I raise this in a seperate forum?

Thank you

Trudi

MiaSrebrnjak
Community Manager

Hi @TMorris7,

 

Thank you for the additional information. I would like to send this to our content tool team for further investigation - could you send me links to your client's pages in a direct message for more context?  

If possible, please include a link to the article where you were able to remove the X-Robots tag by removing and re-adding it to the campaign. 

Thank you!

Mia, Community Team 


Wusstest du, dass es auch eine DACH-Community gibt?
Nimm an regionalen Unterhaltungen teil, indem du deine Spracheinstellungen änderst !


Did you know that the Community is available in other languages?
Join regional conversations by changing your language settings !