Hiding pages from COS Site Search

[AndreaD]

Is there any way to tell the COS Site search to ignore certain pages (or a whole directory?) I have a ton of similarly named landing pages tied to PPC campaigns, download pages that are supposed to be gated, and hidden pages that are only accessible under certain circumstances. I want to know how to hide these from the search results page.

[tjoyce]

@AndreaD

There's 2 ways you can do this:

1. add this meta tag to the head of your page. If you are in the page editor, click Settings -> Advanced Settings and drop this code in the "Head HTML" section:

<meta name="robots" content="noindex,nofollow" />

2. To set it on the global level, got to Content Settings -> Web Pages -> Templates; then look for the Robots.txt section and add the code to disallow the page you want to prevent.

User-agent: * 
Disallow: /index_test
Disallow: /products/test_product
Disallow: /products/ 

BTW, I'm not sure that this will be an immediate fix since google will probably need to reindex your site in order for these changes to take effect so, give it a little time. Also, It wouldn't hurt to combine option 1 and option 2 together. So, go ahead and try them both.

---

tim@belch.io | https://forms.belch.io | Design your HubSpot; No coding necessary

[MFrankJohnson]

@tjoyce, this works for HubSpot COS Site Search results as well?

[tjoyce]

@MFrankJohnson - Good question. Since HS doesn't index LP's by default to be used in the Site Search, I thought @AndreaD may have been referring to the site search generated by google. @AndreaD If you in fact seeing LP's in your native site search, you may either have created those as Web Pages instead or added them to the sitemap manually. I suspect, removing them from the sitemap (if they are there) would probably remove them from the site search. Keep in mind, HubSpot controls the generation of the sitemap and it doesn't change on "Save". I'm also not sure when they regenerate that file but, this may also take some time. I have not tested my theory that the sitemap controls the site search but it seemed the most logical approach they may have taken to the Site Search programming. Let us know the results of your find and we can comment further.

---

tim@belch.io | https://forms.belch.io | Design your HubSpot; No coding necessary

[MFrankJohnson]

Thanks @tjoyce
Makes sense considering that we'd typically set gated, and hidden pages that are only accessible under certain circumstances to "noindex,nofollow", so .... Let's just hope HubSpot follows this logic. Also, very interested in hearing results.

Will test ourselves later this/next week if nothing new posted here by then. Thanks again for your brain power on this. 🙂

Best,
Frank

[GSantos]

Hi @tjoyce

My Landing Pages are not on the sitemap and they are showing in the COS Site Search module, be it the search dropdown as the visitor start typing, or in the page results.

This is really bad, we have no idea for how long this is happening, we have a lot of hidden pages for campaigns that should not be listed.

[AndreaD]

Thank you both for your responses. In fact, I was talking about the COS site search, not Google, but I didn't realize that the HS COS Search didn't do landing pages by default. It also seems not to do blog posts by default, although it is doing HubDB, at least. I think I may need to go back to my old search function until the Hubspot one is more mature.