topic Re: Which directories to block in robots.txt? in Blog, Website & Page Publishing

Which directories to block in robots.txt?

flipflopflower3 — Mon, 20 Jan 2025 17:28:08 GMT

Hello,

Hoping for some advice on best practices regarding allowing/disallowing of images, videos, etc.

In many of our blog posts, we will be adding a video. The videos will be hosted on HubSpot and uploaded to my HubSpot files - then added to a blog post. These are not YouTube videos, so I do not want them to be indexed and served up in search results. I only want a visitor to see the video if they land on a blog post that contains one.

Question: How do I ensure the folder containing the videos is not crawled/indexed? Do I add this to my robots.txt in HubSpot settings? Or is there a different way? Also, are there any downsides to SEO rankings if a blog post is indexed, but the video on it is blocked?

How do you all decide what to include in your robots.txt? I know to use no follow in the head of an actual page I do not want indexed/crawled, but in this example, I do want the page to be crawled.

Thanks in advance for your help!

Re: Which directories to block in robots.txt?

Jnix284 — Mon, 20 Jan 2025 19:10:34 GMT

Hi @flipflopflower3 the best way to manage this would be in your file manager, setting the videos to Public - no index will allow external parties to view them, but keep them out of search.

You can do this per file, when you have the preview pane open on the right:

Or you can do this per folder, open the folder, select all files in the folder, then click "change file visibility" and make your updates on the right side pane:

Re: Which directories to block in robots.txt?

flipflopflower3 — Mon, 20 Jan 2025 19:43:56 GMT

Thank you for your reply! But if I set videos to public-no index, can I still use advanced features? I don't care about putting a form or cta in a video, but I do want to be able to see how a contact is engaging with a video.

Re: Which directories to block in robots.txt?

Jnix284 — Mon, 20 Jan 2025 19:54:24 GMT

@flipflopflower3 yes, you should be able to use the video features the same regardless, this only prevents them from being indexed.

Re: Which directories to block in robots.txt

flipflopflower3 — Mon, 20 Jan 2025 23:07:56 GMT

Thank you again for this information! Can you share best practices for the types of folders or pages that should go into robots.txt?

For example, if a blog post has images in it, is it best practice to make those images public, and allow the folder within HubSpot files that contains the images be indexed/crawled?

If I have downloadable content that a visitor can only access after they have submitted a form (and they will get to download that content from a thank-you page displayed after form submittal), should I mark the actual page no-follow AND ALSO put the url of the folder containing the downloadable files into robots.txt?

Re: Which directories to block in robots.txt

Jnix284 — Tue, 21 Jan 2025 13:23:48 GMT

@flipflopflower3 typically you would include your landing page URLs in the robots.txt and other website pages you don't want followed.

I have not come across a use case where files have been added to robots.txt in HubSpot, especially since they have their own visibility permissions.

Here are the recommended steps to prevent content from appearing in search using HubSpot.