Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Hi, I'd really appreciate advice from anyone who has experienced and overcome the following.
We have geographic IP redirects on our UK .com site which divert US and Canadian users back to our US site. Because of this hubspot can't crawl the UK website for SEO unless we turn our redirects off.
We've contacted hubspot direct with no luck resolving the issue but taken their advice to add their crawler to our robots.txt file and enabled the 'Do not redirect bots and crawlers' function as well as the 'Skip redirection if noredirect=true found in URL' function. Hubspot still can't crawl the site.
The only thing that we have left to try is to exclude IP addresses, like we do for google bots but hubspot say they cannot provide IP ranges for us to allowlist.
The site scans fine when we turn the redirects off but it's a pain to have to do that every 4-6 weeks so we can stay on top of our SEO properly through hubspot.
Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Hi there! Sorry for such a late response! Can I first ask why you decided to re-direct geo users to a different website? Also, how did you setup the re-directs?
Lastly, I think a better option for this would be to use Cloudflare as I believe they have a service like this, but it happens as a WAF, therefore, it just serves a different version of your website instead of actually creating re-directs. Without knowing more, it sounds like you set them up manually, and that could be your issue because if you have a LOT of re-directs, it could simply be timing out.
Also, if you dig deep into a Google search you can definitely find directories that list HubSpot's IP ranges 🤐😎
Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Hi there! Sorry for such a late response! Can I first ask why you decided to re-direct geo users to a different website? Also, how did you setup the re-directs?
Lastly, I think a better option for this would be to use Cloudflare as I believe they have a service like this, but it happens as a WAF, therefore, it just serves a different version of your website instead of actually creating re-directs. Without knowing more, it sounds like you set them up manually, and that could be your issue because if you have a LOT of re-directs, it could simply be timing out.
Also, if you dig deep into a Google search you can definitely find directories that list HubSpot's IP ranges 🤐😎
Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Hi David and thanks for coming back to me and for the tip RE Hubspot IP's, we'll definitely give that a go.
We have a simple redirect on as we have a team in the US who deal with all projects that come in from US IP's, Canadian IP's and Mexican IP's. It's so much easier than having to forward those enquiries as we'd project manage and bill them from the US anyway.
Our web developer has tried all of their recommendations but nothing seems to work, barr turning the redirects off. Hubspot have said they cannot provide thier IP's. They have however run several checks and the crawler is not timing out, it's getting redirected to our US site so our UK site then doesn't get scanned.
I'd potentially be interested in learning how WAF might solve our problem, I understood it as a security function but I'm no expert here. Would you be able to direct me to some (easy to digest) info on this?
Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Just head over to Cloudflare's website and go into the documentation for your specific needs. I don't want to diretcly link to it because I don't want to potentially send you down a path that doesn't work for your use case.
Hubspot can't crawl my site for SEO because of IP redirects
SOLVE
Thanks Kristen and all those that you reached out to. Any advice will be greatly appreciated. I have another SEO crawler issue on another site to deal with too if any of you are interested?