V3 & V4 API-s: Reading metadata/data for associations in an efficient manner

[skerdi]

Hello,

I am a developer of an application which consumes the V3 & V4 API-s.

My use case is to read all association labels data for all objects in the V3 & V4 API-s using the requests below:

Spoiler

GET /crm/v3/objects/contacts?limit=100&associations=companies,tickets,deals,quotes,tasks&archived=false HTTP/1.1

GET /crm/v3/objects/companies?limit=100&associations=contacts,tickets,deals,quotes,tasks&archived=false HTTP/1.1
...

GET /crm/v4/objects/contacts?limit=100&associations=companies,tickets,deals,quotes,tasks&archived=false HTTP/1.1

GET /crm/v4/objects/companies?limit=100&associations=contacts,tickets,deals,quotes,tasks&archived=false HTTP/1.1
...

There are 2 problems that I am facing with this scenario.

1 - The first problem, is that I have to first fetch the object names/type ids, in order to provide the list of values to the associations query parameter. To retrieve this data, I have to make a call to the /schemas/{object} endpoint. I do this by saving every value for the toObjectTypeId property in the label definitions in a set data structure.

Spoiler

GET https://api.hubapi.com/crm/v3/schemas/contact HTTP/1.1

{

    "labels": { "singular": "Contact", "plural": "Contacts" },

    ...

    "associations": [

      {

        "fromObjectTypeId": "0-1",

        "toObjectTypeId": "0-11",

        "name": "CONTACT_TO_CONVERSATION",

        "cardinality": "ONE_TO_MANY",

        "inverseCardinality": "ONE_TO_MANY",

        "hasUserEnforcedMaxToObjectIds": false,

        "hasUserEnforcedMaxFromObjectIds": false,

        "maxToObjectIds": 50000,

        "maxFromObjectIds": 50000,

        "id": "62",

        "createdAt": null,

        "updatedAt": null

      },

      {

        "fromObjectTypeId": "0-11",

        "toObjectTypeId": "0-1",

        "name": "CONVERSATION_TO_CONTACT",

        "cardinality": "ONE_TO_MANY",

        "inverseCardinality": "ONE_TO_MANY",

        "hasUserEnforcedMaxToObjectIds": false,

        "hasUserEnforcedMaxFromObjectIds": false,

        "maxToObjectIds": 50000,

        "maxFromObjectIds": 50000,

        "id": "61",

        "createdAt": null,

        "updatedAt": null

      },

      {

        "fromObjectTypeId": "0-1",

        "toObjectTypeId": "0-118",

        "name": "CONTACT_TO_PAYMENT_LINK",

        "cardinality": "ONE_TO_MANY",

        "inverseCardinality": "ONE_TO_MANY",

        "hasUserEnforcedMaxToObjectIds": false,

        "hasUserEnforcedMaxFromObjectIds": false,

        "maxToObjectIds": 50000,

        "maxFromObjectIds": 50000,

        "id": "470",

        "createdAt": null,

        "updatedAt": null

      }

      ...

  }

In the example above: <0-11, 0-118, etc>.

2 - The second problem (although I guess this is very specific to my use case), is that I want to be able to provide the users with the singular name of the associated object type for every association label. For example:

Spoiler

GET /crm/v3/objects/contacts?limit=100&associations=companies,tickets,deals,quotes,tasks&archived=false HTTP/1.1

GET /crm/v4/objects/contacts?limit=100&associations=companies,tickets,deals,quotes,tasks&archived=false HTTP/1.1

Spoiler

{

  "results": [

    {

      "id": "103381403171",

      ...

    },

    {

      "id": "105363718492",

      ...

      "associations": {

        "courses": {

          "results": [{ "id": "437892522595", "type": "contact_to_course" }]

        },

        "companies": {

          "results": [

            { "id": "21960462656", "type": "contact_to_company" },

            { "id": "21960462653", "type": "contact_to_company_unlabeled" },

            { "id": "21960462656", "type": "contact_to_company_unlabeled" },

            { "id": "21960462657", "type": "contact_to_company_unlabeled" }

          ]

        },

        "p45447734_my_custom_object_2": {

          "results": [

            { "id": "29664938156", "type": "contact_to_my_custom_object_2" }

          ]

        }

      }

    },

    ...

  ]

}

Spoiler

Id=105363718492
AssociatedObjectId=437892522595
AssociatedObjectType=Course
LabelName=contact_to_course

Id=105363718492
AssociatedObjectId=21960462656
AssociatedObjectType=Company
LabelName=contact_to_company

...

However, the API returns the names of the associated objects in plural:

Spoiler

"companies": {
"courses": {

, so I have to make a request to the /schemas/{object} endpoint for every object and build a map of singular-plural object names internally, which I can later use to create the results for the application's users:

Spoiler

{
    "labels": { "singular": "Contact", "plural": "Contacts" }
    ...
}

This means that my users will have to wait some time (call the /schemas/{object} endpoint several times and parse the data), before they are able to read any associations data.

The extra API calls to the /schemas/{objectType} make the overall user experience of the application bad. I think it would be much more convenient and efficient for HubSpot's API consumers if the API provided objects metadata (where associations metadata is also included) in 1 call:

Spoiler

GET https://api.hubapi.com/crm/v3/schemas HTTP/1.1

GET /crm/v3/objects/contacts?limit=100&associations=companies,tickets,deals,quotes,tasks&archived=false HTTP/1.1
GET /crm/v3/objects/companies?limit=100&associations=contacts,tickets,deals,quotes,tasks&archived=false HTTP/1.1

...

I investigated whether this was possible and made some tests, but it seems the /schemas endpoint only returns metadata for user-defined HubSpot objects:

- https://community.hubspot.com/t5/APIs-Integrations/API-endpoint-to-retrieve-a-list-of-all-objects-i...

- https://community.hubspot.com/t5/APIs-Integrations/Object-Schemas-GET-All-Custom-and-Standard/m-p/7...

- https://developers.hubspot.com/docs/reference/api/crm/objects/schemas

Is there any other way through which I can read associations and objects metadata for all objects in 1 request? If not, has HubSpot considered this functionality? In that case, what would be a rough estimate about the time when it's generally released to the public?

Note

If the server was able to detect this use case, by letting the consumer provide some special input/parameter, that would be an even better solution for the first problem. For example, a wildcard:

Spoiler

GET /crm/v3/objects/contacts?limit=100&associations=*&archived=false HTTP/1.1

GET /crm/v3/objects/companies?limit=100&associations=*&archived=false HTTP/1.1
...

GET /crm/v4/objects/contacts?limit=100&associations=*&archived=false HTTP/1.1
GET /crm/v4/objects/companies?limit=100&associations=*&archived=false HTTP/1.1

...

 

Note

In the examples I have provided here, I have used the plural object names in the list of values for the associations query parameter. In reality however, I do use the object type ids as I explained before.

[sylvain_tirreau]

Hi @skerdi ,

Unless I'm mistaken, you can retrieve all object schemas (standard and custom), singular/plural labels and associations via:

GET /crm-object-schemas/v3/schemas?includeStandard=true

And all association definitions/labels via:

GET /crm/v4/associations/definitions/configurations/all

[skerdi]

@sylvain_tirreau , thanks for the suggestions. Regarding what you said:

1. I tested the /crm-object-schemas/v3/schemas endpoint with the includeStandard=true parameter, but just like the /crm/v3/schemas endpoint. only the schemas of custom/user-defined objects are returned.

2. The /crm/v4/associations/definitions/configurations/all endpoint returns the association schemas. I concluded this from tests I made and the API documentation which seems very clear about this. What I want however, is to read the actual associations (instances) that a HubSpot object has with all other HubSpot objects which are associated with it (association labels are defined for these relationships).

[sylvain_tirreau]

1. Does your application grant read permissions for the objects whose associations you want to obtain:

       crm.schemas.<object>.read

?

2. I haven't tested it, but I'll try to do it this way.

With /crm-object-schemas/v3/schemas?includeStandard=true, you retrieve all object metadata. You extract the singular and plural labels for each objectTypeId.

With /crm/v4/associations/definitions/configurations/all, you retrieve all association definitions (labels, etc.). You then construct an object like this:

• assoc_by_id -> direct access to the definition from the associationTypeId.
• from_to_index -> list of possible associationTypeIds between two objects.

Finally, read the instances via the batch API:

POST /crm/v4/associations/{fromObject}/{toObject}/batch/read

[skerdi]

1. Yes, my access token has been granted many of the crm.schemas.<object>.read scopes (e.g. contacts, companies, deals, etc.). However, this endpoint still returns only the schemas of user-defined HubSpot objects.

2. I suppose this is one of the many ways to read associations data. Another less efficient way would be to use the /crm/v4/objects/{objectType}/{objectId}/associations/{toObjectType} endpoint. However, to me both of these solutions are not acceptable. I was trying to explain this in the post's description, but the main idea here is that I need to provide my users with the best performance possible for reading associations data.

- /crm/v4/objects/{objectType}/{objectId}/associations/{toObjectType}, will return all the associations that one object in an object category has with all objects in another object category.

- /crm/v4/associations/{fromObject}/{toObject}/batch/read, will return all the assocaitions that multiples objects in an object category have with all objects in another object category.

- /crm/v4/objects/contacts?associations=companies,tickets,..., will return all the associations that all objects in an object category have will all objects in all object categories with which they are associated with.

When I look at it from this viewpoint, it becomes clear to me that the 3rd option will result in far less API calls/requests, which means better performance in reading all the associations.

Furthermore, the /crm/v4/associations/definitions/configurations/all endpoint seems to return only the schemas of user-defined association labels. The endpoint description in the API documentation kind of hints this as well:

Returns all user configurations available on a given portal

HubSpot's system-defined labels such as CONTACT_TO_COMPANY, CONTACT_TO_DEAL, CONTACT_TO_TICKET, etc. are not returned. Either way, considering what I said regarding performance, I still don't see how this endpoint could help me.

[sylvain_tirreau]

On reflection, I think it's normal that there is only the list of associations customized by the user, given that the other associations are a fixed list, possibly updated from time to time. If I were you, I would store this list and update it regularly. I would do this by querying the endpoints useful for this on a regular basis via a cron. You have a list here, but I think that querying the right endpoints regularly would allow you to have both the customized associations and the associations defined by Hubspot. And this way you achieve significantly improved performance, because you store the data with the structure you want.

[skerdi]

Sorry, but I still do not understand how the association ids can help. It's not like I can use these ids in the associations query parameter:

https://api.hubapi.com/crm/v4/objects/contacts?limit=100&associations=279%2C4%2C15&archived=false

I tried this and it does not work.

[sylvain_tirreau · Accepted solution]

Hi @skerdi ,

I'm sorry, I didn't go into enough detail and probably gave you the wrong link.

1. Locally, you build a dictionary (object in js; I call it here "objects") that will store the singular and plural.

For custom objects, use /crm-object-schemas/v3/schemas.

For standard objects, use /crm-object-schemas/v3/schemas/{object_id}

I would do something like this (in Python) for standards:

standard_ids = {
    "0-1": "Contacts",
    "0-2": "Companies",
    "0-3": "Deals",
    "0-5": "Tickets",
    "0-53": "Quotes",
    "0-18": "Line items",
    "0-27": "Tasks",
    "0-4": "Notes",
    "0-48": "Meetings",
    "0-106": "Calls",
}

for object_id in standard_ids:
    if object_id in objects:
        continue
    r = requests.get(f"https://api.hubapi.com/crm-object-schemas/v3/schemas/{object_id}", headers=HEAD, timeout=15)
    ...

You can build your object with, for example, this structure:

{object_id: {"singular": "...", "plural": "..."}}

2. Then you generate all possible pairs of your object IDs stored in your dictionary (see "itertools.permutations" in Python to achieve the same result with another language or with your own function). In Python, it's a generator, so we can iterate directly on the function, and we can retrieve each possible pair.

For each of the pairs, you retrieve the metadata via /crm/v4/associations/{from}/{to}/labels that you store on your side (in a "associations" dictionnary). You save for example "objects" and "associations" in a json file ({"objects": {...}, "associations": {...}).

You perform those steps once a day, whenever you want.

In your application, you load your cached JSON file and use it as needed.

You probably need to adapt all of this to your environment...

[BérangèreL]

Hi @skerdi and Happy Friday!

Thanks for asking the HubSpot Community and for the detailed information!

I'd love to put you in touch with our Top Experts: Hi @coldrickjack, @sylvain_tirreau and @Anton do you know of a way to read associations and objects metadata for all objects in 1 request to help @skerdi, please?

Have a lovely weekend and thank you very much for your help!
Bérangère