Getting All Contacts with Data Factory

Highlighted
New Contributor

I am trying to get all Contacts into a SQL DB with Data Factory. When I put in the Pagination rule.

Name                                           Value
QueryParameters.vidOffset          $.vid-offset

The GET loops forever because the $.vid-offset becomes 0 when $.has-more is false.

How do I make it stop when All contacts are read.

 

PS. I have used the Hubspot connector but It does not let me specify properties, so I have to do it another way.

 

Edit: Issue still not resolved. Will ask Azure support for help.

Reply
0 Upvotes
2 Replies 2
Highlighted
HubSpot Employee

Looking through the Data Factory pagination documentation here and this article on JSONPath syntax, I believe you substitute the following value:

 

 

$[?(@.has-more==true)].vid-offset

 

 

When "has-more" is false, it will return an empty array.

 

You can confirm by running this path agains the example response from the endpoint documentation. using this tool: http://jsonpath.herokuapp.com/

 

The Data Factory REST connector will stop once "any of the JSONPath expressions in "paginationRules" returns null". I hope the above expression qualifies, but I cannot test it myself.

 

Highlighted
New Contributor

@vbrech Thanks for the response. I tried using the value provided by you, but it only returns 1 page. When I used a vid-offset value that is very large it did indeed stop returning values unlike before, which it just starts from vid-offset=0 again. So I believe we are on the right track.

 

Edit: I did some testing with http://jsonpath.herokuapp.com/

It seems 

$[?(@.has-more==true)].vid-offset

returns [10] while  $.vid-offset returns 10. I think Data Factory only accept integers instead of array. I will play around to see if I can convert the [10] into 10.

Reply
0 Upvotes