So, this has been implemented now, much to my chagrin 
. My company actually relied on the ability to show limited deal information on a deal association card even though a user didn’t have access to the full deal record.
I posted a new idea that HubSpot should create a new permission setting that would allow admins to configure permissions for displaying association cards independent of the permission to view the full object record. I’d appreciate it if you’d give an up-vote to my suggestion.