Using GET /crm/v3/objects/<object> endpoints for extracting associations

[nmfoss]

Hi again,

I am currently creating a API connector to extract data from HubSpot to a datawarehouse solution.

We are using the GET /crm/v3/objects/<object> endpoints to do most of the extraction. Here we also ask for all associations for each object to any object. These associations also come with separate pagination links. We handle this.

My question is: we see that when running a full load using these endpoints we end up missing a bunch of associations. We are certain we are picking up all associations on "first page" (together with the object properties) and all subsequent pagination pages for each type of association.

When we have identified some missing associations we can make a call to the associations endpoint, eg:

GET api.hubapi.com/crm/v4/objects/companies/xxxxx/associations/emails

and here we can find the missing associations. So there is no scoping or user settings restricting the associations.

Is there somekind of undocumented features of how the associations works in the objects endpoints that we are using? For instance there is no documentation on the pagination for these associations, so is there something else we should be aware of that might explain the missing associations?

Sincerely,

NM Foss

[PCronin9 · Accepted solution]

Hi @nmfoss, I agree with you, this is a great question, and I have it too! .If you have an object with more than 100 associations, there _used_ to be an endpoint in the v3 Associations API that would facilitate you paginating the remainder of the associations, but it appears that endpoint has been removed from the documentation and crippled (but not removed!) in functionality (see details in my linked post).

For our organization, we don't have an object with over 100 associations yet, so I haven't had to cross this bridge yet. But, if I had to solve it today, I would use the V4 Associations API (either the Batch - Read or Basic - List endpoints) to get all the associations I need. Do note however that the pagination information from the V3 CRM API is not compatible with the V4 Associations API, so if you do this, I'd throw out the ones you got on the "first page" of results from the V3 CRM API (or simply not request them at all), and use the V4 Associations API to get them all.

For what it's worth, I find this behavior of the HubSpot APIs quite alarming, and I find the lack of participation, comment, and interest in bug reports from anyone employed by HubSpot to be simiarly alarming. In my experience working with the HubSpot APIs that I do, the developer experience has been quite weak, and I have not found anyone at HubSpot yet who seems to share my concerns with what appear to be genuine bugs in their APIs.

[GKurtulmus]

I use the following format to be able to retrieve all the data for a given object:

function.basic_api.get_page(
limit=100,
properties=properties,
associations = associations,
archived=False,
after=after)

For this purpose I use pagination with the value at ''response.paging.next.after'' and add it to the 'after'.

But if there are associations that specifically require pagination, there are new keys for each of the association type. Named as
'associations.contacts.paging.next.after' or

'associations.companies.paging.next.after'.

How should I query them to get all the associations for the given record?

[PCronin9 · Accepted solution]

Hi @nmfoss, I agree with you, this is a great question, and I have it too! .If you have an object with more than 100 associations, there _used_ to be an endpoint in the v3 Associations API that would facilitate you paginating the remainder of the associations, but it appears that endpoint has been removed from the documentation and crippled (but not removed!) in functionality (see details in my linked post).

For our organization, we don't have an object with over 100 associations yet, so I haven't had to cross this bridge yet. But, if I had to solve it today, I would use the V4 Associations API (either the Batch - Read or Basic - List endpoints) to get all the associations I need. Do note however that the pagination information from the V3 CRM API is not compatible with the V4 Associations API, so if you do this, I'd throw out the ones you got on the "first page" of results from the V3 CRM API (or simply not request them at all), and use the V4 Associations API to get them all.

For what it's worth, I find this behavior of the HubSpot APIs quite alarming, and I find the lack of participation, comment, and interest in bug reports from anyone employed by HubSpot to be simiarly alarming. In my experience working with the HubSpot APIs that I do, the developer experience has been quite weak, and I have not found anyone at HubSpot yet who seems to share my concerns with what appear to be genuine bugs in their APIs.

[BérangèreL · Accepted solution]

Hi @PCronin9 and @nmfoss,

Thanks for your valuable feedback. This means a lot to us.
I have shared it internally for more visibility.

I'm sorry to hear about your experience regarding this.

I would like to apologize for the frustration, data issues, and business impact this has caused.

Thank you @PCronin9 also for sharing this workaround that can help in the meanwhile.

I have checked internally for you and I could see that HubSpot will soon be releasing a V4 of the CRM Object API that will help for this.
I don't have an exact timeline to share for this at the moment but the Team is actively working on it as we speak.

Thank you for your patience.

Have a great day!

Best,
Bérangère

[EMårtensson]

Hi there, we are facing this issue to. Any update on the v4 API? "Soon" was 8 months ago! 🙂 

[PCronin9]

Is there a reason the associations API is not a sufficient solution for your use case?

[EMårtensson]

Hi PCronin:

By now Ive spent enough time on this issue to work around it. Whenever we encounter a company with a next pagination, I refetch all associations from scratch using the Associations v4. Just cumbersome and tough to explain to others looking at the repo why it has to be that way. But yes, its possible to work around and its not that much more labor intensive than having a working pagination (as we only miss the first page when starting anew).

However, the amount of time spent debugging that "later" value (assuming it worked and realizing after some hours that it didnt) was a painful experience. Having undocumented behavior like this in the API is a pain when developing. So I raise any issues I see to validate the price tag for Enterprise that we're happily paying 🙂 

[BérangèreL]

Hi @nmfoss,

Thank you for asking the Community!

I believe there may be a limit in the number of rows you can get. Which API documentation did you use, please? Is there any limit mentioned on it?

Have you tried breaking up your call in several requests to get all associations, please?

I also wanted to invite a couple of subject matter experts and some Community Members to this conversation: Hi @alyssamwilie, @jonchim, @PCronin9, @HBrandao and @stefen do you have suggestions to help @nmfoss, please?

If anybody else has anything to add and/or share, please feel free to join in the conversation 🙂

Thanks a lot and have a wonderful day!

Best,
Bérangère