Contacts defaults to "Contact Owner"

I understand that, but everytime I refresh the page it defaults to Contact Owner & it automatically applies the filter… How do I eliminate that, where it defaults to my contacts. I even edit the Filter but it always defaults to that… Super frustrating that every time I switch between tasks and contacts I have to either click the “X” or edit the filter.