Marketing Integrations

DChapman
Participant

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi, I'd really appreciate advice from anyone who has experienced and overcome the following.

 

We have geographic IP redirects on our UK .com site which divert US and Canadian users back to our US site. Because of this hubspot can't crawl the UK website for SEO unless we turn our redirects off.

 

We've contacted hubspot direct with no luck resolving the issue but taken their advice to add their crawler to our robots.txt file and enabled the 'Do not redirect bots and crawlers' function as well as the 'Skip redirection if   noredirect=true found in URL' function. Hubspot still can't crawl the site.

 

The only thing that we have left to try is to exclude IP addresses, like we do for google bots but hubspot say they cannot provide IP ranges for us to allowlist.

 

The site scans fine when we turn the redirects off but it's a pain to have to do that every 4-6 weeks so we can stay on top of our SEO properly through hubspot.

 

Can anyone shed light on this please?

 

Thanks

2 Accepted solutions
DavidDennison
Solution
Key Advisor

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi there! Sorry for such a late response! Can I first ask why you decided to re-direct geo users to a different website? Also, how did you setup the re-directs?

 

Lastly, I think a better option for this would be to use Cloudflare as I believe they have a service like this, but it happens as a WAF, therefore, it just serves a different version of your website instead of actually creating re-directs. Without knowing more, it sounds like you set them up manually, and that could be your issue because if you have a LOT of re-directs, it could simply be timing out.

 

Also, if you dig deep into a Google search you can definitely find directories that list HubSpot's IP ranges 🤐😎

David Dennison

Search Engine Optimization

and Content Marketing Expert

mobilePhone
702-556-5062
emailAddress
david@daviddennison.com
website
https://daviddennison.com

View solution in original post

Jnix284
Solution
Most Valuable Member | Diamond Partner
Most Valuable Member | Diamond Partner

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi @DChapman I'm not sure if you went with the Cloudfare option, but I'm curious why instead of redirects you aren't using smart rules to change the content based on country?


If my reply answered your question please mark it as a solution to make it easier for others to find.



Jennifer Nixon - Delivery Lead at Aptitude 8

connect with Jen on Linkedin

View solution in original post

6 Replies 6
Jnix284
Solution
Most Valuable Member | Diamond Partner
Most Valuable Member | Diamond Partner

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi @DChapman I'm not sure if you went with the Cloudfare option, but I'm curious why instead of redirects you aren't using smart rules to change the content based on country?


If my reply answered your question please mark it as a solution to make it easier for others to find.



Jennifer Nixon - Delivery Lead at Aptitude 8

connect with Jen on Linkedin

DavidDennison
Solution
Key Advisor

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi there! Sorry for such a late response! Can I first ask why you decided to re-direct geo users to a different website? Also, how did you setup the re-directs?

 

Lastly, I think a better option for this would be to use Cloudflare as I believe they have a service like this, but it happens as a WAF, therefore, it just serves a different version of your website instead of actually creating re-directs. Without knowing more, it sounds like you set them up manually, and that could be your issue because if you have a LOT of re-directs, it could simply be timing out.

 

Also, if you dig deep into a Google search you can definitely find directories that list HubSpot's IP ranges 🤐😎

David Dennison

Search Engine Optimization

and Content Marketing Expert

mobilePhone
702-556-5062
emailAddress
david@daviddennison.com
website
https://daviddennison.com
DChapman
Participant

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi David and thanks for coming back to me and for the tip RE Hubspot IP's, we'll definitely give that a go.

 

We have a simple redirect on as we have a team in the US who deal with all projects that come in from US IP's,  Canadian IP's and Mexican IP's. It's so much easier than having to forward those enquiries as we'd project manage and bill them from the US anyway.

 

Our web developer has tried all of their recommendations but nothing seems to work, barr turning the redirects off. Hubspot have said they cannot provide thier IP's. They have however run several checks and the crawler is not timing out, it's getting redirected to our US site so our UK site then doesn't get scanned.

I'd potentially be interested in learning how WAF might solve our problem, I understood it as a security function but I'm no expert here. Would you be able to direct me to some (easy to digest) info on this?

 

Best

 

Dawn

DavidDennison
Key Advisor

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Just head over to Cloudflare's website and go into the documentation for your specific needs. I don't want to diretcly link to it because I don't want to potentially send you down a path that doesn't work for your use case.

David Dennison

Search Engine Optimization

and Content Marketing Expert

mobilePhone
702-556-5062
emailAddress
david@daviddennison.com
website
https://daviddennison.com
DChapman
Participant

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Thanks Kristen and all those that you reached out to. Any advice will be greatly appreciated. I have another SEO crawler issue on another site to deal with too if any of you are interested?

 

kvlschaefer
Community Manager
Community Manager

Hubspot can't crawl my site for SEO because of IP redirects

SOLVE

Hi @DChapman,

 

Thanks for reaching out!

 

I'll add some experts to our conversation to see if they have any insight.

Hi @danmoyle@Sam62@DavidDennison@himanshurauthan - Are you aware of a workaround or do you have advice for  @DChapman

 

Thanks!

 

Best,
Kristen


Did you know that the Community is available in other languages?
Join regional conversations by changing your language settings !