robots.txt

[JaganPrasath]

Does all the webdevelopment tools have robots.txt in them like html, javascript, etc., or it only exist in html?

I read it somewhere that html file is good for robots.txt file. Kindly  clarify it.

[Jnix284 · Accepted solution]

Hi @JaganPrasath a robots.txt file is a text file, it's not HTML, JS, etc. just plain-text.

HubSpot includes one automatically that you can update via settings > website > pages > SEO & crawlers:

[JaganPrasath]

And one more thing is, only the pearson having the access to the C-panel account can modify or upload a robots.txt file. (Robots.txt: The Deceptively Important File All Websites Need (hubspot.com) i read it in this llink which was shared by you

[Jnix284]

In HubSpot, it requires access to "Website Settings" which doesn't require C-panel account access:

Which can be found in User Settings here: