Hello! Conversations API product manager here. It is true that there is not currently an endpoint to set the contact associated with a conversation.
That said, what we really want to invest in here is our own automated contact assignment logic instead of leaning on the developer community to patch it up. It’s hard for me to know from this post whether some of the examples of this you are experiencing are ‘working as designed’ (e.g. there is not currently a contact in HubSpot with a phone number that matches the one on the incoming conversation) or not. This might sound like an obvious research question, but, “why is it important for you to be able to automatically assign a contact to these conversations instead of ask a user to do it manually so that the data quality will be checked by them?”