Don’t lose your mind! It’s not easy/obvious (they need to work on that), but you can filter on contacts with labels on deals (I don’t know about custom object).
