Thanks @Kaleudia, good to know (though frustrating).
I would be ok with restricting access by team if I could assign all these contacts to myself (which I’ve done) and put myself in a team (done) and then be able to set a permission that only people in my team can see my contacts. Is there a way to do that without putting all of our Hubspot contacts in another team?