Delay API email events

[MarkdeLange]

We have created a custom integration between Hubspot and an end point in Veeva. Veeva is a SalesForce solution created for the pharmaceutical industry. We are having issues with the number of email events we see in Hubspot and Veeva. After deeper analysis:

• We know that there is a delay to pass data to the email event API. Perhaps this is more likely to happen when a number of large emails are sent, triggering many delivered and sent email events.
• We use the offset token on subsequent requests on the email events endpoint with the same query parameters. [see script below]
• We believe it may be due to the delay to get all the email events in the API, when the offset token does not enable us to find the last record which synched.
• As a result we are getting significant variations in the number of email (sent/delivered/opened/clicked) events in Hubspot and the Veeva
• We can synch an individual email which does not the timestamp using the same code. When we do this we the email events pass correctly.
• We use the EPOC timestamp.

Below is an example of the code we use.

$startTimestamp = strtotime("-5 minutes"); // {now - 5 minutes};

$paramsO = "&startTimestamp=".$startTimestamp. "000"; 

$limit = "&limit=1000";

$params = "$paramsO$limit";

$hasmore = true;

$offset = false;

$events = [];

while($hasmore){

                if($offset !== false) $params = "$paramsO$limit&offset=$offset";

                $camp = $this->callAPI($api,$parametros);

                $ch = curl_init();

                $url = "https://api.hubapi.com/email/public/v1/events{hubspottoken}".$params;

                curl_setopt($ch, CURLOPT_URL,$url);

                curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

                curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);

                $response = curl_exec($ch);

                curl_close($ch);

                $response = json_decode($response);

                $hasmore = $response->hasMore;

                $offset = urlencode($response->offset);

                $events = array_merge($events, $response->events);

}

[malcolm1]

Assuming that there aren't unique problems within individual emails that are causing problems. That's a nightmare. Assuming that the email content is deliverable, have you tried batching the process? Lots of small packets can succeed when one big one doesn't.

[dennisedson]

@malcolm1 , @Mike_Eastwood , maybe you all could take a look at this 🙏

[dennisedson]

@MarkdeLange 

Thanks for sharing the code.  @himanshurauthan , do you see anything unusual that would be causing the issue?