# modify x-robots-tag: none from directory **URL:** \ **Category:** Content Strategy & SEO\ **Tags:** free\ **Created:** [July 14, 2025, 5:30am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147 "2025-07-14T05:30:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![IshuSingh94](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/ishusingh94/32/207849_2.png) [@IshuSingh94](https://community.hubspot.com/u/IshuSingh94)\ **Post date:** [July 14, 2025, 5:30am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/1 "2025-07-14T05:30:05Z")
How to change and modify “x-robots-tag: none” from a file or directory to indexing the page easily in search engines ? ---
**Author:** ![karstenkoehler](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/karstenkoehler/32/68554_2.png) [@karstenkoehler](https://community.hubspot.com/u/karstenkoehler)\ **Post date:** [July 14, 2025, 5:36am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/2 "2025-07-14T05:36:01Z")
Hi @IshuSingh94, Let’s leverage generative AI to answer this query: > * * * > > ### 1. Determine Where the Header Is Being Set > > The `X-Robots-Tag` HTTP header might be configured in: > > - Apache `.htaccess` or main `httpd.conf` configuration > - NGINX `nginx.conf` or site-specific config > - Server-side code (PHP, Node.js, Python, etc.) > - Web hosting control panels (such as cPanel or Plesk) > - CDN or firewall layers (like Cloudflare or Sucuri) > - Application frameworks or CMS platforms > > Use a tool like `curl -I` or an online HTTP header checker to identify which files or directories return this header. > > ### 2. Modify the Configuration Based on Your Environment > > **Apache (via `.htaccess` or `httpd.conf`)** > > Search for a line that sets the `X-Robots-Tag` header. If it is set to “none”, you will need to either remove the line or change it to “index, follow”. After modifying `httpd.conf`, make sure to restart Apache. > > **NGINX** > > In the server block or location block of your configuration, locate any directive setting `X-Robots-Tag` to “none”. You can remove this line or change it to “index, follow”. After editing, reload the NGINX configuration. > > **PHP or Other Application Code** > > If the header is being set in server-side code, locate and modify or remove the function that adds the `X-Robots-Tag: none` header. You can also explicitly set it to “index, follow” to ensure indexing is allowed. > > **Cloudflare or Other CDN Providers** > > Check the response header rules within your CDN dashboard. If there is a rule applying `X-Robots-Tag: none` to specific URLs or file types, remove it or change the value to “index, follow”. > > **robots.txt File** > > While not related to the `X-Robots-Tag` header directly, ensure your `robots.txt` file is not disallowing the directory or file in question. A `Disallow` rule for the path will prevent search engines from crawling it regardless of headers. > > ### 3. Confirm That the Header Has Been Updated > > After making changes, test the URL again using a header inspection tool or by running a curl command. The response should no longer contain `X-Robots-Tag: none`. If present, it should now read `index, follow`, or be entirely absent, which by default allows indexing. > > * * * Hope this helps! ---
**Author:** ![IshuSingh94](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/ishusingh94/32/207849_2.png) [@IshuSingh94](https://community.hubspot.com/u/IshuSingh94)\ **Post date:** [July 14, 2025, 6:23am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/3 "2025-07-14T06:23:42Z")
how to change the hubspot default website ---
**Author:** ![karstenkoehler](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/karstenkoehler/32/68554_2.png) [@karstenkoehler](https://community.hubspot.com/u/karstenkoehler)\ **Post date:** [July 14, 2025, 6:25am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/4 "2025-07-14T06:25:51Z")
@IshuSingh94 I’m not sure what exactly you’re referring to. Could you please share a bit more context? If you’re referring to the default page you’re seeing when accessing the app, you can set that under Settings → General → Profile. ---
**Author:** ![IshuSingh94](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/ishusingh94/32/207849_2.png) [@IshuSingh94](https://community.hubspot.com/u/IshuSingh94)\ **Post date:** [July 14, 2025, 6:57am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/5 "2025-07-14T06:57:07Z")
I want to index my HubSpot website URL, [https://toptechtarget-243220195.hs-sites-na2.com/toptechtarget](https://toptechtarget-243220195.hs-sites-na2.com/toptechtarget), in a search engine. but got a error for indexing and crawling in PageSpeed Insights. ![Capture_error.PNG](https://d2qpr2jzcazej7.cloudfront.net/original/3X/1/4/143f2fec8c40ca753cef0e4a20be1595d8a5b74b.png) ## how to access this file or directory to modify these changes ## ---
**Author:** ![karstenkoehler](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/karstenkoehler/32/68554_2.png) [@karstenkoehler](https://community.hubspot.com/u/karstenkoehler)\ **Post date:** [July 14, 2025, 7:07am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/6 "2025-07-14T07:07:40Z")
@IshuSingh94 could you confirm the following? - Settings \> Content \> Pages \> SEO & Crawlers - Scroll down to “Custom `Robots.txt`” - Make sure your path (`/toptechtarget`) is not disallowed (e.g., `Disallow: /toptechtarget` should not be listed) Also, can you confirm your page itself is not set to no-index? [Prevent content from appearing in search results](https://knowledge.hubspot.com/seo/prevent-content-from-appearing-in-search-results) ---
**Author:** ![Aakar](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/aakar/32/107785_2.png) [@Aakar](https://community.hubspot.com/u/Aakar)\ **Post date:** [July 14, 2025, 9:41am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/7 "2025-07-14T09:41:20Z")
Hi @IshuSingh94 - You can set up this setting on both a domain level and a page level. If you want to control it at the domain level, go to Settings \> Content \> Pages \> SEO & Crawlers. I would advise editing or updating this if you understand how it works. I’d suggest checking HubSpot’s robots.txt to get some ideas: [https://www.hubspot.com/robots.txt](https://www.hubspot.com/robots.txt) If you’re thinking of indexing your website, ensure you’ve setup sitemap.xml as well, which you can access from: Settings \> Content \> Domains & URLs \> Sitemap Thanks. ---
**Author:** ![IshuSingh94](https://d10pqhm3lwygmk.cloudfront.net/user_avatar/community.hubspot.com/ishusingh94/32/207849_2.png) [@IshuSingh94](https://community.hubspot.com/u/IshuSingh94)\ **Post date:** [July 14, 2025, 11:31am UTC](https://community.hubspot.com/t/modify-x-robots-tag-none-from-directory/136147/8 "2025-07-14T11:31:49Z")
Hi @aakarpost, Thanks for helping me out to successfully submit my sitemap. ![solutions.PNG](https://d2qpr2jzcazej7.cloudfront.net/original/3X/6/6/66f31497d6447016bc963416202427f667eb96bf.png) toptechtarget sitemap