https://community.hubspot.com/t5/APIs-Integrations/HubSpot-Links-Crawler-2-0-user-agent-hitting-protected-pages/m-p/229546#M8442 <P><A class="mention" href="https://community.hubspot.com/u/derek_gervais">@Derek_Gervais</A> Yes, they have been implemented since day 1 as the page path is accessing an authenticated account. Here is what is looks like:</P> <PRE><CODE class="lang-auto"># See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-agent: * Disallow: / User-agent: AdsBot-Google Disallow: / </CODE></PRE> Tue, 27 Feb 2018 15:46:08 GMT terrance 2018-02-27T15:46:08Z