Restrict access to conversations in the shared inbox, similar to standard object restrictions

Following this as well. I want to collaborate with our sales team in our billing shared inbox but I do NOT want them to be able to reply to the customers using our AR@ email address. Someone did that by accident from a completely different ticket pipeline and it was super weird.