modify x-robots-tag: none from directory

[IshuSingh94]

How to change and modify "x-robots-tag: none" from a file or directory to indexing the page easily in search engines ?

[karstenkoehler · Accepted solution]

@IshuSingh94 could you confirm the following?

• Settings > Content > Pages > SEO & Crawlers

• Scroll down to “Custom Robots.txt”

• Make sure your path (/toptechtarget) is not disallowed (e.g., Disallow: /toptechtarget should not be listed)

Also, can you confirm your page itself is not set to no-index? https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results

[Aakar · Accepted solution]

Hi @IshuSingh94 - You can set up this setting on both a domain level and a page level. If you want to control it at the domain level, go to Settings > Content > Pages > SEO & Crawlers. I would advise editing or updating this if you understand how it works.

I'd suggest checking HubSpot's robots.txt to get some ideas: https://www.hubspot.com/robots.txt

If you're thinking of indexing your website, ensure you've setup sitemap.xml as well, which you can access from: Settings > Content > Domains & URLs > Sitemap

Thanks.

[Aakar · Accepted solution]

Hi @IshuSingh94 - You can set up this setting on both a domain level and a page level. If you want to control it at the domain level, go to Settings > Content > Pages > SEO & Crawlers. I would advise editing or updating this if you understand how it works.

I'd suggest checking HubSpot's robots.txt to get some ideas: https://www.hubspot.com/robots.txt

If you're thinking of indexing your website, ensure you've setup sitemap.xml as well, which you can access from: Settings > Content > Domains & URLs > Sitemap

Thanks.

[IshuSingh94 · Accepted solution]

Hi @aakarpost, Thanks for helping me out to successfully submit my sitemap.

toptechtarget sitemap

[karstenkoehler]

Hi @IshuSingh94,

Let's leverage generative AI to answer this query:

---

1. Determine Where the Header Is Being Set

The X-Robots-Tag HTTP header might be configured in:

• Apache .htaccess or main httpd.conf configuration

• NGINX nginx.conf or site-specific config

• Server-side code (PHP, Node.js, Python, etc.)

• Web hosting control panels (such as cPanel or Plesk)

• CDN or firewall layers (like Cloudflare or Sucuri)

• Application frameworks or CMS platforms

Use a tool like curl -I or an online HTTP header checker to identify which files or directories return this header.

2. Modify the Configuration Based on Your Environment

Apache (via .htaccess or httpd.conf)

Search for a line that sets the X-Robots-Tag header. If it is set to "none", you will need to either remove the line or change it to "index, follow". After modifying httpd.conf, make sure to restart Apache.

NGINX

In the server block or location block of your configuration, locate any directive setting X-Robots-Tag to "none". You can remove this line or change it to "index, follow". After editing, reload the NGINX configuration.

PHP or Other Application Code

If the header is being set in server-side code, locate and modify or remove the function that adds the X-Robots-Tag: none header. You can also explicitly set it to "index, follow" to ensure indexing is allowed.

Cloudflare or Other CDN Providers

Check the response header rules within your CDN dashboard. If there is a rule applying X-Robots-Tag: none to specific URLs or file types, remove it or change the value to "index, follow".

robots.txt File

While not related to the X-Robots-Tag header directly, ensure your robots.txt file is not disallowing the directory or file in question. A Disallow rule for the path will prevent search engines from crawling it regardless of headers.

3. Confirm That the Header Has Been Updated

After making changes, test the URL again using a header inspection tool or by running a curl command. The response should no longer contain X-Robots-Tag: none. If present, it should now read index, follow, or be entirely absent, which by default allows indexing.

---

Hope this helps!

[IshuSingh94]

how to change the hubspot default website

[karstenkoehler]

@IshuSingh94 I'm not sure what exactly you're referring to. Could you please share a bit more context?

If you're referring to the default page you're seeing when accessing the app, you can set that under Settings → General → Profile.

[IshuSingh94]

I want to index my HubSpot website URL, https://toptechtarget-243220195.hs-sites-na2.com/toptechtarget, in a search engine. but got a error for indexing and crawling in PageSpeed Insights.

how to access this file or directory to modify these changes

 

[karstenkoehler · Accepted solution]

@IshuSingh94 could you confirm the following?

• Settings > Content > Pages > SEO & Crawlers

• Scroll down to “Custom Robots.txt”

• Make sure your path (/toptechtarget) is not disallowed (e.g., Disallow: /toptechtarget should not be listed)

Also, can you confirm your page itself is not set to no-index? https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results