API Stream to Bigquery with no rate limits on calls

[Imartinezdetoda]

Hi there!

I'm trying to set up an integration between hubspot and google cloud where i want to create a cloud function which feeds from the GET API calls that hubspot provides, to upload this information into my bigquery tables.

I'm having issues due to the rate limit of 100 requests per call given that i have more than 9K registers on my companies table.

Does someone has experienced the same issue? how can i build a workaround to ingest all my data via these API calls and have it updated withouth having to worry about the rate limits?

Thanks in advance!!

[johnelmer · Accepted solution]

@Imartinezdetoda I suspect this use case is what HubSpot had in mind with Operations Hub Enterprise, specifically the Snowflake data share. https://knowledge.hubspot.com/reports/query-hubspot-data-in-snowflake

[Jaycee_Lewis]

Hi, @Imartinezdetoda 👋 Thanks for the question. Let's see if we can get the conversation going. Hey, @JBeatty @johnelmer @Mike_Eastwood, have you tackled similar projects? Or do you have any thoughts on how @Imartinezdetoda might approach this challenge is other ways? 

Best,

Jaycee

[Imartinezdetoda]

Hi Jaycee, thanks for bringing awareness to the question!

I've actually created a workaround for using the multiple API call results and concatenate the results to have a final json file to use for streaming my data into Bigquery.

But it is not scalable due to computational load, API rate limits, and the data volume we plan to use in in the future.

Any thoughts on how to tackle this and create a streaming pipeline to Bigquery for all my hubspot data will be much appreciated.

Thanks in advance team!! 

[Eric_Goldman]

Hi @Imartinezdetoda we're working on some solutions to this exact problem at Sequin. I'm curious to learn more about your use case and see if we might be able to help. Send me an email, DM, or start a chat on our site → sequin.io, and we can find a time to chat.