So it looks like the accepted solution does not work in 2022, but I managed to do the following in Python API:
from hubspot.crm.schemas import ObjectTypeDefinitionPatch
object_patch = ObjectTypeDefinitionPatch(primary_display_property='my_property', required_properties=['my_property'], searchable_properties=['my_property'])
hubspot_client.crm.schemas.core_api.update(object_type=MY_OBJECT_TYPE, object_type_definition_patch=object_patch)
So the idea is to reset all the parts of the object schema where the old column is used (primary_display_property, required_properties, and searchable_properties in my case)
)