Questions regarding activities sync

[dmaurer]

Objective

Our company wants to ingest data from HubSpot into an on premise, centralized data repository so that data from HubSpot can be combined with other systems. Collection of data in a central location will allow the data team to create analytic solutions which have a wider view of customer interactions which are collected in multiple source systems including Microsoft Dynamics (CRM) and Oracle (ERP).

The data of interest from HubSpot includes customer interaction / activity data (e.g. emails, forms, notes). The business request is that all activity data is collected and made available for use by the data lake.

Background

Our company operates another CRM (Microsoft Dynamics) which is responsible for governing certain aspects of the sales workflow that are not propagated to HubSpot. In most cases, integration between HubSpot and Dynamics is one way (from HubSpot -> Dynamics). There is some data which only exist in HubSpot and is not propagated to Dynamics.

Solution Approach

We are aware of the API reference (HubSpot APIs | Getting started) and have reviewed the available endpoints. We found that the request to “get all activity feed information” can be said differently as “for each type of activity in the HupSpot activity feed, get all instances for that activity type from its respective endpoint using the HupSpot API.” It appears that each activity type has its own endpoint. Within each endpoint, HupSpot offers the ability to specify properties which are returned in the API response in addition to the default properties.

To begin development, we will need to agree on a list of endpoints from which we want to collect data. For each endpoint, we will need to also define which properties we want to include in the extraction. We will then develop a process which orchestrates requests for multiple endpoints, iteration if necessary, and potentially generating additional requests based on results of previous response data.

Questions

• Is this the approach in line with the best practices recommended by HubSpot? Is there any strategy that they recommend for customers with similar data objectives?
• Does HubSpot offer a schema which describes the relationship of objects and engagements? This would help us have a better understanding of how to organize the data as it lands in our repository.
• Is the request to use the existing APIs to collect data from the Activity feed feasible? Are the events / activity types which are available in the user interface but not currently made available in the API?

[Jaycee_Lewis]

Hi, @dmaurer 👋 Thanks for reaching out. Let's see if we can get the conversation going — hey @Phil_Vallender @louischausse @himanshurauthan do you have any big picture advice you can offer to @dmaurer?

Best,

Jaycee