Crawl anomaly on all my website pages

Johanne
Member

Hello, 

 

We just released our new website on hubspot, but unfortunatly our pages can't be indexed by google. When we try to see them as googlebot on the search console, we have a crawl anomaly (https://cl.ly/c8b189ec7594), it seems to be a 403. 

I found this article talking about this issue, saying : "Googlebot: HubSpot does not allow the crawling of HubSpot pages from the Googlebot originating from non-Google IP addresses. If you attempt to crawl your HubSpot site as Googlebot, you will likely see a 403 error."

 

But I don't see any solution to this problem? What can I do to make sure my website can be indexed by google ? I can't event submit a sitemap on the search console as Google can't "fetch" it.  

 

Thanks a lot, 

Johanne

0 Upvotes
5 Replies 5
MatthewShepherd
HubSpot Employee

Hi @Johanne 

Can you share your domain URL so I can take a look at your robots.txt and page source code?


Matthew Shepherd

Senior Inbound Consultant

Professional Services | HubSpot

He/Him

linkedin.com/in/matthewshepherd/
https://www.hubspot.com/services/professional
0 Upvotes
Johanne
Member

Yes sure, it's https://www.comet.co/. The homepage is not on Hubspot, but all the others pages are (https://www.comet.co/pourquoi-comethttps://www.comet.co/fonctionnaliteshttps://www.comet.co/offre-entrepriseshttps://www.comet.co/offre-startupshttps://www.comet.co/tarifs ...).

 

I already checked at the robot.txt on hubspot, and I searched for a noindex in the source code but maybe I missed something (I hope so 🙂 )

 

Thanks a lot, 

0 Upvotes
MatthewShepherd
HubSpot Employee

Hi @Johanne 

I can crawl the Hubspot pages and yes they are indexable, and I don't see any issues in the robots.txt file, but when I crawl your home page my text-based crawler doesn't find the other pages of your site.  I think JavaScript could be the issue here.

If you crawl your site with a text-only crawler your site contains very little useful HTML (page content) and no links to allow the crawler to find your Hubspot pages. Google can render and index Javascript sites, but they aren't the best idea for SEO.

Google has indexed the home page of your site, it might just take some time for it to render your JavaScript and find the next links to crawl.

 

comet-index.png

 

From an SEO standpoint, I would try and serve more of your site's essential content without needing JavaScript to render it. If that's not possible I would check for JavaScript errors - if you check the page through Google's Mobile-Friendly tool (Page loading issues->VIEW DETAILS) you will see there are some errors which could be causing crawling and rendering issues.

comet-issues.png

 

From there I would follow Google's advice on fixing search related JavaScript issues.

 

Hope that helps.


Matthew Shepherd

Senior Inbound Consultant

Professional Services | HubSpot

He/Him

linkedin.com/in/matthewshepherd/
https://www.hubspot.com/services/professional
0 Upvotes
Johanne
Member

Thank you very much for your fast answer Matthiew!

However, I'm pretty sure it's not the problem : our homepage is in javascript but can can actually be read by the googlebot (https://cl.ly/83c6766dbed4) > it's the only page indexed for now. 

 

The problem is for all the others pages (that are hosted on hubspot and not coded in javascript), for instance : 

https://www.comet.co/pourquoi-comet

https://www.comet.co/offre-entreprises

 

The google bot is unable to read them (403 error when I make some crawler tests), so we can't ask for an indexation, so the pages can't be indexed... Same for the hubspot sitemap that can't be submitted in the search console (https://cl.ly/5c45ac335a6a). 

 

Thanks again,

0 Upvotes
MatthewShepherd
HubSpot Employee

OK, good to know.

 

In that case, I can't see any other issues that could be causing this on your side. I'd recommend submitting a ticket to the Hubspot support team so they can check what is happening.


Matthew Shepherd

Senior Inbound Consultant

Professional Services | HubSpot

He/Him

linkedin.com/in/matthewshepherd/
https://www.hubspot.com/services/professional
0 Upvotes