@SRickman the permissions are set by who owns the contact, company, etc., so if you have different sales reps working with the same company, it won’t be possible to exclude activity from other reps, but you can do this for notes.
Have you already reviewed their individual permissions?