May 25, 20229:02 AM - edited May 25, 202212:48 PM
Member
Extracting Contacts data via API
SOLVE
Hi
We currently extract contacts data via API using Azure Data Factory - just delta changes.
However, due to batch limit of 100, the extract takes a long time (days) especially if we try an extract soon after a bulk update to most of the contacts, as we have in excess 300k contacts .
Is there an alternative method available that we can use that is quicker?
We've got a recursive function that walks batches of 100 Contacts (with a 150ms sleep to avoid throttling)... unfortunately there's no real way to avoid it.
We've got a recursive function that walks batches of 100 Contacts (with a 150ms sleep to avoid throttling)... unfortunately there's no real way to avoid it.
What sort of performance do you get from the function? Do you have a similar number of contacts? It would help to know if we can get the runtime from days down to hours or to possibly minutes to help us prioritise. Creating an Azure function is an option that we're considering. The alternative API endpoint will definately help.