Extracting Contacts data via API

[MisterM]

Hi

We currently extract contacts data via API using Azure Data Factory - just delta changes.

However, due to batch limit of 100, the extract takes a long time (days) especially if we try an extract soon after a bulk update to most of the contacts, as we have in excess 300k contacts .

Is there an alternative method available that we can use that is quicker? 

[Mike_Eastwood · Accepted solution]

Hi @MisterM 

We've got a recursive function that walks batches of 100 Contacts (with a 150ms sleep to avoid throttling)... unfortunately there's no real way to avoid it. 

Can you use the Get recently updated and created contacts endpoint? So you you're only working on recently change Contacts?

Or, could you create an App with a Webhook that makes changes in real time as Contacts change?

Curiosly

Mike

p.s. thanks for the mention @dennisedson 

[MisterM]

Hi @Mike_Eastwood 

Thankyou for replying.

What sort of performance do you get from the function?  Do you have a similar number of contacts?  It would help to know if we can get the runtime from days down to hours or to possibly minutes to help us prioritise. 
Creating an Azure function is an option that we're considering.
The alternative API endpoint will definately help.

[dennisedson]

@Mike_Eastwood , any thoughts on this pickle?