Show contact notes separately
It's useful to see & scroll through the contact timeline (emails, tasks, calls..) but contact Notes shouldn't be lost in that pile, as they are not necessarily specific to a moment in time, but are more general or contain key info to be seen before calling the contact.
Notes should be listed separately and prominently (maybe above the timeline in an expandable box?). If notes can be added by different users, this notes display should be filterable by author. They should be right there, visible without clicking or scrolling, in the same way that email and phone numbers should be right there. This (in my experience) is normal practice in other CRMs.
To be more generic, maybe you could have two timeline windows (both with a much more compact layout - see suggestions elsewhere) which can each have their own filters. The filter choice remains until explicitly changed and can be saved as a view layout. That way each user can customise the contact view to give them the data they need.