Currently, users with no user management permissions can still see all the teams (including members) and permission sets (including settings and permissions).
This is not cool as it means that people can find out who else can do things they can’t, causing potential conflicts and being a general security issue.
Instead, there should be another permission (again under Account Access) for ‘View team members’ and ‘View permission sets’.