Blog, Website & Page Publishing

flipflopflower3
Participant

Which directories to block in robots.txt?

SOLVE

Hello,

Hoping for some advice on best practices regarding allowing/disallowing of images, videos, etc.

 

In many of our blog posts, we will be adding a video. The videos will be hosted on HubSpot and uploaded to my HubSpot files - then added to a blog post. These are not YouTube videos, so I do not want them to be indexed and served up in search results. I only want a visitor to see the video if they land on a blog post that contains one.

 

Question: How do I ensure the folder containing the videos is not crawled/indexed? Do I add this to my robots.txt in HubSpot settings? Or is there a different way? Also, are there any downsides to SEO rankings if a blog post is indexed, but the video on it is blocked?

 

How do you all decide what to include in your robots.txt? I know to use no follow in the head of an actual page I do not want indexed/crawled, but in this example, I do want the page to be crawled.

 

Thanks in advance for your help!

0 Upvotes
1 Accepted solution
Jnix284
Solution
Hall of Famer

Which directories to block in robots.txt?

SOLVE

Hi @flipflopflower3 the best way to manage this would be in your file manager, setting the videos to Public - no index will allow external parties to view them, but keep them out of search.

 

You can do this per file, when you have the preview pane open on the right:

Jnix284_0-1737400102952.png

 

Or you can do this per folder, open the folder, select all files in the folder, then click "change file visibility" and make your updates on the right side pane:

Jnix284_1-1737400207667.png

 


If my reply answered your question please mark it as a solution to make it easier for others to find.


Jennifer Nixon

View solution in original post

0 Upvotes
5 Replies 5
Jnix284
Solution
Hall of Famer

Which directories to block in robots.txt?

SOLVE

Hi @flipflopflower3 the best way to manage this would be in your file manager, setting the videos to Public - no index will allow external parties to view them, but keep them out of search.

 

You can do this per file, when you have the preview pane open on the right:

Jnix284_0-1737400102952.png

 

Or you can do this per folder, open the folder, select all files in the folder, then click "change file visibility" and make your updates on the right side pane:

Jnix284_1-1737400207667.png

 


If my reply answered your question please mark it as a solution to make it easier for others to find.


Jennifer Nixon
0 Upvotes
flipflopflower3
Participant

Which directories to block in robots.txt?

SOLVE
Thank you for your reply! But if I set videos to public-no index, can I still use advanced features? I don't care about putting a form or cta in a video, but I do want to be able to see how a contact is engaging with a video.
0 Upvotes
Jnix284
Hall of Famer

Which directories to block in robots.txt?

SOLVE

@flipflopflower3 yes, you should be able to use the video features the same regardless, this only prevents them from being indexed.


If my reply answered your question please mark it as a solution to make it easier for others to find.


Jennifer Nixon
0 Upvotes
flipflopflower3
Participant

Which directories to block in robots.txt?

SOLVE
Thank you again for this information! Can you share best practices for the types of folders or pages that should go into robots.txt?

For example, if a blog post has images in it, is it best practice to make those images public, and allow the folder within HubSpot files that contains the images be indexed/crawled?

If I have downloadable content that a visitor can only access after they have submitted a form (and they will get to download that content from a thank-you page displayed after form submittal), should I mark the actual page no-follow AND ALSO put the url of the folder containing the downloadable files into robots.txt?
0 Upvotes
Jnix284
Hall of Famer

Which directories to block in robots.txt?

SOLVE

@flipflopflower3 typically you would include your landing page URLs in the robots.txt and other website pages you don't want followed.

 

I have not come across a use case where files have been added to robots.txt in HubSpot, especially since they have their own visibility permissions.

 

Here are the recommended steps to prevent content from appearing in search using HubSpot.


If my reply answered your question please mark it as a solution to make it easier for others to find.


Jennifer Nixon
0 Upvotes