Missed API calls to AWS

gepot
Contributor

I developed a serverless app on AWS Lambda that removes contacts who were added via CC without consent.  It works well, but I noticed that not all contact events trigger the Lambda function.

 

I can see that Hubspot fires off the Contact Create event under the webhooks monitor, but when I look at the log files on AWS those contact IDs don't appear.  I keep a list of contacts that were skipped, so I can manually run them through the procedure, but this isn't ideal.

 

I changed the API call rate for the Hubspot app to try and mitigate the issue, but alas, there are still contacts that don't get picked up.

 

I am not sure where to look for the issue as it seems to disappear between Hubspot and AWS.  Any ideas?

0 Upvotes
5 Replies 5
WendyGoh
HubSpot Employee

Hi @gepot,

 

Are you seeing that when a contact is being created, the webhooks monitoring will show the event however these contacts ain't being write into the log files on AWS?

 

Just to be sure, are you referring to the Manage CCed Contacts app? When looking at the webhooks monitoring, I'm able to see that there's a number of 502 error (filtered by status: failure).

Can you check if these are the missing contacts? 

gepot
Contributor

Hi @WendyGoh ,

Thanks for looking into this!

 

Yes, the app is Manage CCed Contacts.  I am aware of the 502 errors and they were resolved, and I can see those errors on my AWS log files anyway (including the contact ID that caused the problem). 

 

You are correct in your understanding - I see in the webhooks monitor that a given contact was created (I can see the contact ID), but when I go into my AWS logs, that contact is nowhere to be seen.  I generate detailed logs referencing the contact ID for just this purpose.

 

What makes this odd is that it only happens intermittently - so far I failed to see a pattern. It is as if the webhook event fires off, but it never reaches the Lambda function.

 

I keep track of these failed attempts in a dynamic list where the Mark for Review property is unknown (I use this to mark poor quality contacts, but every new contact has this field set when they are parsed through the Manage CCed Contacts app).

 

 

0 Upvotes
WendyGoh
HubSpot Employee

Hey @gepot,

 

That’s odd. I would expect that if the webhook notifications is showing in-app on the webhooks monitor, the AWS logs should contain the notification.

 

In this case, can I check if you're logging the incoming HTTP request or if you're just logging the objectId in your application code? If it's the latter, it might be that the HTTP requests are coming in but something is up between accepting the request and the execution of your lambda code?

 

Additionally, another probable explanation may be due o how Lambda handles concurrency. So maybe your lambda concurrency settings are too low, and some requests are getting dropped/lost or being throttled.

 

Hope his helps!

0 Upvotes
gepot
Contributor

Hi @WendyGoh 

Thanks for your inputs.  I updated my Lambda function to log the request ID in the handler itself, before calling the function that manages the CCed contacts, but alas, I again sit with contacts that weren't processed and no sign of any calls making it to the Lambda function.

 

I also checked my concurrency settings - default is 1000 calls per minute which should be ample.  The last two contacts that were processed were many minutes apart, and the only two contacts processed today, so concurrency doesn't seem to be the issue.

 

All I know is that Hubspot fires off the webhook (otherwise it won't appear in the monitor logs) and the call seemingly randomly doesn't make it to AWS.  Here is what a typical log entry would look like n AWS for this function:

 

 

lambda-log.png

 

I am at a loss Smiley Frustrated

 

Edit:  I increased the timeout setting to see if I can catch any failures.  I will report back.

0 Upvotes
mansoor292
Member

There is a gotcha in Lambda. I am not sure of the extent of your problem and you probably already solved it.

 

A lambda will stop executing as soon as a response is sent back. So you need to hold it open for either an aribtrary amount of time or wait for a callback form hubspot before returning a response. 

 

0 Upvotes