Create a permanent deleted (HTTP code 410) for a lot of URLs

[JSGasse]

We used to host our website on another service and encountered issues with them. While we were in the process of transferring the site to HubSpot, our website went down, and the company we were doing business with could not bring it back. As a result, we have a lot (and I mean a lot) of URLs that no longer exist. Some have new URLs, but most we no longer have at all.

While we may be able to redo them over time, we want to ensure that Google no longer indexes them and that we only index the new pages on the new URLs. I submitted a "temporary page removal" request for "URLs starting with," but even with that, Google still attempts to index them once the removal is lifted. The only way I found to remove them permanently is to set the server to return a 410 code for those pages.

Is there a way to configure HubSpot to return a 410 code for all URLs that start with example.com/blogs?

[Aakar]

@JSGasse - I have no idea whether HubSpot supports 410. But how about redirecting (301) all of those indexed URLs (but non-existent) to your new website/blog/page? You can import URLs in bulk and redirect them to your new pages in HubSpot. Here's the HubSpot article.

Thanks.

[JSGasse]

The problem is that those pages will still appear in Google Search. When I check Google Search Console, I see 1,500 indexing attempts per day on pages that no longer exist (404), and Google continues to attempt to index them. According to the forum, the only way to permanently de-index a page is to return a 410 Gone (permanently deleted) code; only then will Google remove the page. Even if some pages return a 404, they still appear in Google searches. Thanks for the help anyway.

[Aakar · Accepted solution]

If you're already moving all of it or some of it to new website, then redirecting to the new page won't harm. If they appear in search engine, they will be redirected to new page.

But if you want to deindex all of it, you can disallow them in robots.txt. How easy/difficult for you to update/manage robots.txt file? If they follow similar kind of URL pattern, let's say /blog. You can easily disallow these links in robots.txt

[kennedyp]

Good workaround @Aakar -- here is a relevant Knowledge Base article on preventing content from being indexed: https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results

[kennedyp]

Hi @JSGasse! Welcome to the Community-- happy to have you here 😊

I understand your goal is to keep your old pages, but temporarily disable them and keep them from indexing until you can resolve any issues. Is there content on those pages currently or is the goal just to get those individual URLs with no content removed from the index? 

Maybe our experts can help! Hey @coreysmith, @Crystal_Hopper, or @benM can you provide any guidance here? Thanks! 

Best, 
Kennedy