Crawl anomaly on all my website pages

Highlighted
Occasional Contributor

Hello, 

 

We just released our new website on hubspot, but unfortunatly our pages can't be indexed by google. When we try to see them as googlebot on the search console, we have a crawl anomaly (https://cl.ly/c8b189ec7594), it seems to be a 403. 

I found this article talking about this issue, saying : "Googlebot: HubSpot does not allow the crawling of HubSpot pages from the Googlebot originating from non-Google IP addresses. If you attempt to crawl your HubSpot site as Googlebot, you will likely see a 403 error."

 

But I don't see any solution to this problem? What can I do to make sure my website can be indexed by google ? I can't event submit a sitemap on the search console as Google can't "fetch" it.  

 

Thanks a lot, 

Johanne

Reply
0 Upvotes
5 Replies 5
Highlighted
Advisor | Platinum Partner

Hi @Johanne 

Can you share your domain URL so I can take a look at your robots.txt and page source code?


Did my post help answer your query? Help the Community by marking it as a solution.

Matthew Shepherd

An Inbound Growth Agency
Platinum HubSpot Partner

Reply
0 Upvotes
Highlighted
Occasional Contributor

Yes sure, it's https://www.comet.co/. The homepage is not on Hubspot, but all the others pages are (https://www.comet.co/pourquoi-comethttps://www.comet.co/fonctionnaliteshttps://www.comet.co/offre-entrepriseshttps://www.comet.co/offre-startupshttps://www.comet.co/tarifs ...).

 

I already checked at the robot.txt on hubspot, and I searched for a noindex in the source code but maybe I missed something (I hope so Smiley Happy )

 

Thanks a lot, 

Reply
0 Upvotes
Highlighted
Advisor | Platinum Partner

Hi @Johanne 

I can crawl the Hubspot pages and yes they are indexable, and I don't see any issues in the robots.txt file, but when I crawl your home page my text-based crawler doesn't find the other pages of your site.  I think JavaScript could be the issue here.

If you crawl your site with a text-only crawler your site contains very little useful HTML (page content) and no links to allow the crawler to find your Hubspot pages. Google can render and index Javascript sites, but they aren't the best idea for SEO.

Google has indexed the home page of your site, it might just take some time for it to render your JavaScript and find the next links to crawl.

 

comet-index.png

 

From an SEO standpoint, I would try and serve more of your site's essential content without needing JavaScript to render it. If that's not possible I would check for JavaScript errors - if you check the page through Google's Mobile-Friendly tool (Page loading issues->VIEW DETAILS) you will see there are some errors which could be causing crawling and rendering issues.

comet-issues.png

 

From there I would follow Google's advice on fixing search related JavaScript issues.

 

Hope that helps.


Did my post help answer your query? Help the Community by marking it as a solution.

Matthew Shepherd

An Inbound Growth Agency
Platinum HubSpot Partner

Reply
0 Upvotes
Highlighted
Occasional Contributor

Thank you very much for your fast answer Matthiew!

However, I'm pretty sure it's not the problem : our homepage is in javascript but can can actually be read by the googlebot (https://cl.ly/83c6766dbed4) > it's the only page indexed for now. 

 

The problem is for all the others pages (that are hosted on hubspot and not coded in javascript), for instance : 

https://www.comet.co/pourquoi-comet

https://www.comet.co/offre-entreprises

 

The google bot is unable to read them (403 error when I make some crawler tests), so we can't ask for an indexation, so the pages can't be indexed... Same for the hubspot sitemap that can't be submitted in the search console (https://cl.ly/5c45ac335a6a). 

 

Thanks again,

Reply
0 Upvotes
Highlighted
Advisor | Platinum Partner

OK, good to know.

 

In that case, I can't see any other issues that could be causing this on your side. I'd recommend submitting a ticket to the Hubspot support team so they can check what is happening.


Did my post help answer your query? Help the Community by marking it as a solution.

Matthew Shepherd

An Inbound Growth Agency
Platinum HubSpot Partner

Reply
0 Upvotes