topic Re: HS Search module showing duplicate blog posts. in CMS Development

HS Search module showing duplicate blog posts.

MHowe — Mon, 22 May 2023 16:57:51 GMT

During the development of one of our sites we used a test blog with real blog posts and we called it "Test Blog". When we launched we either changed the name to "Blog" or created a new blog and removed "Test Blog". When you search for certain blog posts, it will return duplicate posts, one with /blog/ in the url and one with /test-blog/ url that goes to a 404.

There is no instance of "Test Blog" anymore and there are no re-directs still in place. I have this to robots.txt but it still returns when you search:

Disallow: /test-blog/
Disallow: /test-blog/*

What else can I try?

Re: HS Search module showing duplicate blog posts.

Jaycee_Lewis — Tue, 23 May 2023 20:06:32 GMT

Hi, @MHowe 👋 Thanks for your question.

I have a few of my own to help give the community more context:

Are these old URLs are still indexed by Google or another search engine? If so, have you already requested a re-crawl or removal of the obsolete URLs?
Are these duplicates appearing in the HubSpot CMS search or on an external search engine?
Is there any chance that “test-blog” URLs might have been hardcoded anywhere on your site? For example, in blog templates, modules, or elsewhere?

Thank you for the additional information! — Jaycee

Re: HS Search module showing duplicate blog posts.

MHowe — Tue, 23 May 2023 21:10:33 GMT

Thank you for responding Jaycee:

I will look further into this, I am not 100% sure.
Just the HubSpot CMS, more specificaly the native HubSpot search module. When I search in Google only one listing (the correct one) comes up)
There are no hard-coded urls that would include "test-blog"