There is not currently a HubSpot API to get contacts that cross-reference to the owner of that contact. When getting contacts via the API, you will return all contacts. In your external application you would need to build logic to only show contacts that a user should see, based on their combination of view/edit/communicate.