We don’t want any of our knowledge base content indexed in Google, so we have:
1) Turned on the general “noindex” function, which covers articles in the knowledge base
2) Blocked search engines with robots.txt
But category pages are still indexable (and still indexed), and the HubSpot support team told me I can’t put a noindex tag on these pages.
This is highly problematic, especially because HubSpot keeps suggesting the Robots.txt setup as the solution, but Google is clearly against that and only respects a noindex tag. Right now, our main site is competing with our KB site.. it’s a disaster.
So, a suggestion is to include category pages in the “noindex” setup or give us the option to edit the header for each page so that we can do it ourselves.
@Christian_Jorge I wish I had better news, unfortunately what you’re trying to do isn’t possible with the current limitations in place.
I completely agree this functionality is needed, the good news is that there is already an open idea that is In Planning, the bad news is that it has been for a while and was put on hold in January 2022.
With that said, the knowledge base team is actively working on a lot of improvements and monitoring the ideas closely - adding your feedback to the existing idea is the best way to reach them (I added a comment as well).
To clarify for others, the noindex function only adds noindex tags to the “search results” pages, not to articles themselves:
And while you could update your robots.txt as you mentioned, this doesn’t add the noindex tag to each article or page.
Within the article, under Settings > Advanced Options you could add the noindex tag to the Head HTML section:
This would allow you to add it to articles, but as you mentioned it would exclude category pages.
The alternative would be to add it at the domain level, which would block it for all pages (including category pages) for your domain where you have your KB…
However, it would only work if your KB is on a subdomain that is unique and not used for your website or other content…
EXCEPT, HubSpot excludes this change for Knowledge Base, so we’re stuck with not being able to add it to the category pages:
<meta name=“robots” content=“noindex”>
