# Robots.txt issue

**URL:** <https://community.hubspot.com/t/robots-txt-issue/8602>\
**Category:** Reporting & Analytics\
**Created:** [April 23, 2018, 3:10pm UTC](https://community.hubspot.com/t/robots-txt-issue/8602 "2018-04-23T15:10:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rickbyrd](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/rickbyrd/32/52876_2.png) [@rickbyrd](https://community.hubspot.com/u/rickbyrd)\
**Post date:** [April 23, 2018, 3:10pm UTC](https://community.hubspot.com/t/robots-txt-issue/8602/1 "2018-04-23T15:10:26Z")

</div>

The robots.txt file within Hubspot is not matching the robots.txt file in Google Search Console. See images below.

 ![Hubspot_Robots_file.png](https://d2qpr2jzcazej7.cloudfront.net/original/3X/3/2/32a7a672e98c3dc1df45be2083ce69e45e6741be.png)  
Hubspot robots.txt  
 ![GSC_Robots_file.png](https://d2qpr2jzcazej7.cloudfront.net/original/3X/f/9/f9a19137c1dab20a63ed413e1e72b931371b8380.png)  
GSC robots.txt

I do not know how lines 7 - 10 showed up in the GSC version. I do not know how to remove them as they are not showing up in Hubspot.

Any thoughts?

Rick

---

<div class="post-metadata">

**Author:** ![Phil\_Vallender](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/phil_vallender/32/48924_2.png) [@Phil\_Vallender](https://community.hubspot.com/u/Phil_Vallender)\
**Post date:** [April 23, 2018, 4:10pm UTC](https://community.hubspot.com/t/robots-txt-issue/8602/2 "2018-04-23T16:10:33Z")

</div>

Hi @rickbyrd

Those look like they are inserted by HubSpot automatically to prevent your post and page previews, email preferences pages and some other things from showing up in search. This is probably a good thing. Do you have a specific reason for wanting to remove them?

---

<div class="post-metadata">

**Author:** ![rickbyrd](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/rickbyrd/32/52876_2.png) [@rickbyrd](https://community.hubspot.com/u/rickbyrd)\
**Post date:** [April 23, 2018, 5:17pm UTC](https://community.hubspot.com/t/robots-txt-issue/8602/3 "2018-04-23T17:17:18Z")

</div>

[@Phil\_Vallender](https://community.hubspot.com/t5/user/viewprofilepage/user-id/74),

They are showing a blocked resources in GSC so I wanted to get them cleared but there is no real good reason if these resources are blocked. It is mostly the Recent Posts listings that are blocked.

Thanks for the explanation!

Rick

---

<div class="post-metadata">

**Author:** ![mcoley](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/mcoley/32/84874_2.png) [@mcoley](https://community.hubspot.com/u/mcoley)\
**Post date:** [April 23, 2018, 6:12pm UTC](https://community.hubspot.com/t/robots-txt-issue/8602/4 "2018-04-23T18:12:38Z")

</div>

@rickbyrd We block some of these resources because the URL’s are signed in order to prevent an attacker from leveraging the service to DDOS a different website. GC would try to follow old expired links and throw errors, so we decided to just block them entirely since they are not important to the indexing of the page.

---

<div class="post-metadata">

**Author:** ![BRuka](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@BRuka](https://community.hubspot.com/u/BRuka)\
**Post date:** [January 27, 2025, 10:06am UTC](https://community.hubspot.com/t/robots-txt-issue/8602/5 "2025-01-27T10:06:28Z")

</div>

Why is my robots.txt blocking the homepage when I didn’t set it to block?  
Here are the screenshots kindly check:

 ![BRuka_0-1737972281091.png](https://d2qpr2jzcazej7.cloudfront.net/original/3X/e/9/e99bb54b4a11071c344f7827c63f14a7b2615551.png)

 ![BRuka_1-1737972356360.png](https://d2qpr2jzcazej7.cloudfront.net/original/2X/3/36e2545c1dad8646417a0d2fb2e3235d12ecc5d1.png)