When you login using OAuth, you give the plugin permission to do things, eg permission to access contacts. Assuming you say yes to this, the plugin will have your rights within the contact area.
When a user installs your app, HubSpot doesn’t send along their user information. The authentication only occurs when installing your app, and only administrators can install apps: