Yes I understand! You could leave this in “Hubspot Ideas forum”, because it would be very useful if it were feasible. For now, the only way for users to see the contacts assigned to you is for those users to belong to the same team as you, but they will not necessarily be contacts assigned only to you, because they can continue to see contacts assigned to those users. The configuration of access to contacts restricts only view, communication and editing by teams, so everyone would have to belong to the same team as you.
Have a nice day!