https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/933805#M3077 <P>Does all the webdevelopment tools have robots.txt in them like html, javascript, etc., or it only exist in html?</P><P>&nbsp;</P><P>I read it somewhere that html file is good for robots.txt file. Kindly&nbsp; clarify it.</P> Tue, 27 Feb 2024 17:01:53 GMT JaganPrasath 2024-02-27T17:01:53Z https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/933805#M3077 <P>Does all the webdevelopment tools have robots.txt in them like html, javascript, etc., or it only exist in html?</P><P>&nbsp;</P><P>I read it somewhere that html file is good for robots.txt file. Kindly&nbsp; clarify it.</P> Tue, 27 Feb 2024 17:01:53 GMT https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/933805#M3077 JaganPrasath 2024-02-27T17:01:53Z https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/933953#M3078 <P>Hi&nbsp;<a href="https://community.hubspot.com/t5/user/viewprofilepage/user-id/717184">@JaganPrasath</a>&nbsp;a robots.txt file is a text file, it's not HTML, JS, etc. just plain-text.</P> <P>&nbsp;</P> <P>HubSpot includes one automatically that you can update via settings &gt; website &gt; pages &gt; SEO &amp; crawlers:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Jnix284_0-1709064178585.png" style="width: 400px;"><img src="https://community.hubspot.com/t5/image/serverpage/image-id/111619i97F3AB57838F6020/image-size/medium?v=v2&amp;px=400" role="button" title="Jnix284_0-1709064178585.png" alt="Jnix284_0-1709064178585.png" /></span></P> <P>&nbsp;</P> Tue, 27 Feb 2024 20:03:12 GMT https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/933953#M3078 Jnix284 2024-02-27T20:03:12Z https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/934588#M3079 <P>And one more thing is, only the pearson having the access to the C-panel account can modify or upload a robots.txt file. (<A href="https://blog.hubspot.com/marketing/robots-txt-file" target="_blank">Robots.txt: The Deceptively Important File All Websites Need (hubspot.com)</A>&nbsp;i read it in this llink which was shared by you</P> Wed, 28 Feb 2024 15:39:40 GMT https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/934588#M3079 JaganPrasath 2024-02-28T15:39:40Z https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/934723#M3080 <P>In HubSpot, it requires access to "Website Settings" which doesn't require C-panel account access:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Jnix284_0-1709140140285.png" style="width: 400px;"><img src="https://community.hubspot.com/t5/image/serverpage/image-id/111728i926686DD8713E390/image-size/medium?v=v2&amp;px=400" role="button" title="Jnix284_0-1709140140285.png" alt="Jnix284_0-1709140140285.png" /></span></P> <P>&nbsp;</P> <P>Which can be found in User Settings here:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Jnix284_1-1709140198817.png" style="width: 400px;"><img src="https://community.hubspot.com/t5/image/serverpage/image-id/111729i5D9AD5C279C9D92C/image-size/medium?v=v2&amp;px=400" role="button" title="Jnix284_1-1709140198817.png" alt="Jnix284_1-1709140198817.png" /></span></P> <P>&nbsp;</P> Wed, 28 Feb 2024 17:12:05 GMT https://community.hubspot.com/t5/Content-Strategy-SEO/robots-txt/m-p/934723#M3080 Jnix284 2024-02-28T17:12:05Z