APIs & Integrations

MisterM
Member

Extracting Contacts data via API

SOLVE

Hi

We currently extract contacts data via API using Azure Data Factory - just delta changes.

However, due to batch limit of 100, the extract takes a long time (days) especially if we try an extract soon after a bulk update to most of the contacts, as we have in excess 300k contacts .

Is there an alternative method available that we can use that is quicker? 

0 Upvotes
1 Accepted solution
Mike_Eastwood
Solution
Key Advisor | Gold Partner
Key Advisor | Gold Partner

Extracting Contacts data via API

SOLVE

Hi @MisterM 

 

We've got a recursive function that walks batches of 100 Contacts (with a 150ms sleep to avoid throttling)... unfortunately there's no real way to avoid it. 

 

Can you use the Get recently updated and created contacts endpoint? So you you're only working on recently change Contacts?

 

Or, could you create an App with a Webhook that makes changes in real time as Contacts change?

 

Curiosly

Mike

 

p.s. thanks for the mention @dennisedson 

View solution in original post

3 Replies 3
Mike_Eastwood
Solution
Key Advisor | Gold Partner
Key Advisor | Gold Partner

Extracting Contacts data via API

SOLVE

Hi @MisterM 

 

We've got a recursive function that walks batches of 100 Contacts (with a 150ms sleep to avoid throttling)... unfortunately there's no real way to avoid it. 

 

Can you use the Get recently updated and created contacts endpoint? So you you're only working on recently change Contacts?

 

Or, could you create an App with a Webhook that makes changes in real time as Contacts change?

 

Curiosly

Mike

 

p.s. thanks for the mention @dennisedson 

MisterM
Member

Extracting Contacts data via API

SOLVE

Hi @Mike_Eastwood 

 

Thankyou for replying.

 

What sort of performance do you get from the function?  Do you have a similar number of contacts?  It would help to know if we can get the runtime from days down to hours or to possibly minutes to help us prioritise. 
Creating an Azure function is an option that we're considering.
The alternative API endpoint will definately help.

dennisedson
Community Manager
Community Manager

Extracting Contacts data via API

SOLVE

@Mike_Eastwood , any thoughts on this pickle?

Thanks,

Dennis




HubSpot Community Developer ShowMake sure to subscribe to our YouTube channel
where you can find the HubSpot Community Developer Show
0 Upvotes