Hi there, Our server is getting requests to private URLs with a User Agent of: "HubSpot Links Crawler 2.0 http://www.hubspot.com/ " It cannot access these URLs and I understand that it will respect robots.txt if provided. What I'd like
Beitrag ansehen