APIs & Integrations

tclark228
Member

Get recently modified or created contacts bug - Indexing delay

SOLVE

We're using the Get recently modified or created contacts endpoint https://legacydocs.hubspot.com/docs/methods/contacts/get_recently_updated_contacts. The API returns contacts in order of lastmodifieddate (newest first)

We've noticed that this API is returning inconsistent data which appears to be caused by indexing delays. 
We update 25 contacts in Hubspot using the bulk edit functionality

We make a request and the most recently modified contact has a timestamp of 1592488442291, the next contact has a lastmodifieddate of 1592488441522
This seems correct and all is good in the world. However 5 
minutes later we make the same request and the most recently modified contact has a timestamp of 1592488442291 which is the same as before. However this contact is a new one and the 2nd contact in the list is now the first one we saw on our previous request. Additionally there are new contacts showing in the list with older timestamps

 

This appears to make the endpoint unusable as you don't know if you're actually getting all updates. Is this just temporary and possibly a bug on Hubspot's end? Is there an alternative method for polling Hubspot? 

 

 

0 Upvotes
1 Accepted solution
WendyGoh
Solution
HubSpot Employee
HubSpot Employee

Get recently modified or created contacts bug - Indexing delay

SOLVE

Hey @tclark228,

 

Just an update here!

 

At the moment, the Get recently updated and created contacts endpoint reads from ElasticSearch to find contacts with a recent last-modified-date, and returns them in order of their last-modified-date (most recent first).

 

In your case, it looks like contact ID 3051's update was indexed into ElasticSearch a little bit delayed, just after your team had done the first query but before your team had repeated it the second time.

 

While we aim for a <5s index time from the update, the bulk updates may sometime take longer.

View solution in original post

0 Upvotes
6 Replies 6
WendyGoh
HubSpot Employee
HubSpot Employee

Get recently modified or created contacts bug - Indexing delay

SOLVE

Hey @tclark228,

 

On my end, I'm unable to reproduce this behavior in my own portal. 

 

I was able to see the same contact returned as the first in the list when using the Get recently updated and created contacts back to back (5mins delay each time, just like your test). 

 

In order for me to further troubleshoot this, could you share with me the following:

1. The portal ID in question.

2. The first return body and the second return body (after 5 mins) that is showing the differences.

0 Upvotes
tclark228
Member

Get recently modified or created contacts bug - Indexing delay

SOLVE

1st Request

 

{
    "contacts": [
        {
            "addedAt": 1593001672723,
            "vid": 2751,
            "canonical-vid": 2751,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mMj9GoGf1ROxH7HnB5nvCwPIgbiJ2aY0Ni8_oBNN0TyCmrSeF1U94lNcN38UgyMEKTwxbfRNhSYHNPAF6kZomZttuqzAM2gGc2z2s39oCPn3aqqVotNN8l6gNavBePHEciyaEzF",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2751",
            "properties": {
                "firstname": {
                    "value": "Taylor"
                },
                "lastmodifieddate": {
                    "value": "1593001672723"
                },
                "createdate": {
                    "value": "1587466324914"
                },
                "lastname": {
                    "value": "C"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2751,
                    "saved-at-timestamp": 1587466324956,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "mytestuser@mytestuser1234.com",
                            "timestamp": 1587466324914,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "af013e81-8f25-4539-ad8c-0b8ed0c6fd41",
                            "timestamp": 1587466324949
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1593001672492,
            "vid": 2851,
            "canonical-vid": 2851,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mM-vnqEpZ2YMJo_g5bskoyUgKfMBlKZTAwHJOXXvHNfGjb7xSYZl8YpspTRLEllv3WwiFtO1xKgflKmFCK5RgrrLPArCR1LurPTxyLnfwO3eGvSOhFxsSNjL_e2Q2hnlIvB863j",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2851",
            "properties": {
                "firstname": {
                    "value": "Marcel"
                },
                "lastmodifieddate": {
                    "value": "1593001672492"
                },
                "createdate": {
                    "value": "1587754374105"
                },
                "lastname": {
                    "value": "Marceau"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2851,
                    "saved-at-timestamp": 1588724516827,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "LEAD_GUID",
                            "value": "2ee59508-4715-45d3-9df4-cc29c070848d",
                            "timestamp": 1587754374137
                        },
                        {
                            "type": "EMAIL",
                            "value": "taylor+test@tray.io",
                            "timestamp": 1588724516763,
                            "is-primary": true
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1593001671935,
            "vid": 2801,
            "canonical-vid": 2801,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mPvzRpCwvfDTAI4qKn8ZYOrHqiiEuTifmNjJ4fV8SUP57xP4CiA02XjjbTr59yMeZhcannBFNf-siZUqECgbVnl8nSRLluRvsSJGtmyoHZFECO57QsxoYV8RXUuYbqb5QlrnIPW",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2801",
            "properties": {
                "firstname": {
                    "value": "test"
                },
                "lastmodifieddate": {
                    "value": "1593001671935"
                },
                "createdate": {
                    "value": "1587466459996"
                },
                "lastname": {
                    "value": "user"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2801,
                    "saved-at-timestamp": 1587466460019,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "test@testesres.com",
                            "timestamp": 1587466459996,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "52f9fcf7-bfff-4d96-8744-e7e765c4d1b3",
                            "timestamp": 1587466460015
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1593001671742,
            "vid": 2901,
            "canonical-vid": 2901,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mOQj0co1hYiB1cR_b7gIJ1_wLQH47yogbtlFXgjYC-677HnewY6L0leQcKP_Dc2J3HQ1OCbFmNJu-s_q9pVNzLmIoOS1ugtBeYBkqVPkHRijwl0b6CdHUHmRBH5p7TvosHo8G79",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2901",
            "properties": {
                "firstname": {
                    "value": "a"
                },
                "lastmodifieddate": {
                    "value": "1593001671742"
                },
                "createdate": {
                    "value": "1591121431306"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2901,
                    "saved-at-timestamp": 1591121431339,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "LEAD_GUID",
                            "value": "a4925df1-5c8b-450f-bc8e-ef769ca1ce1b",
                            "timestamp": 1591121431330
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592926529538,
            "vid": 3401,
            "canonical-vid": 3401,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mNfwXc9XyOYrmF3CsoH4au5Ow3ZyOGbXaIUz-joB5oDxiIRMaBsP5PtSxYIuoCTHAlhQISENnWqVyNMlUtM1MCjM7mj0bTwEmkMZZbbvyZ8SW9eo0kAj8ienuWjF3OHlx5sS-3q",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/3401",
            "properties": {
                "lastmodifieddate": {
                    "value": "1592926529538"
                },
                "createdate": {
                    "value": "1592926504968"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 3401,
                    "saved-at-timestamp": 1592926505113,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "test@t4st.com",
                            "timestamp": 1592926504968,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "2766879c-8169-4d53-a17c-474768348484",
                            "timestamp": 1592926505083
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592571915044,
            "vid": 2651,
            "canonical-vid": 2651,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mPetqOc09qRkYypOtegOX73w9q7aolLZQz0YpVaF5jXZ6Dcq9C9x2ODJznodgk-7MEYrgESjVMTKDf8DIVvzmyGwV8NhSB2zj8_GDuZIQUmUlQifklqXjb1sUSmu0IfwTmwdHmM",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2651",
            "properties": {
                "firstname": {
                    "value": "Test"
                },
                "lastmodifieddate": {
                    "value": "1592571915044"
                },
                "createdate": {
                    "value": "1587407444767"
                },
                "lastname": {
                    "value": "User2"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2651,
                    "saved-at-timestamp": 1587407444844,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "testuser2@testuser.com",
                            "timestamp": 1587407444767,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "3db1513e-8538-4c46-a931-bc842075f255",
                            "timestamp": 1587407444801
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592571914163,
            "vid": 2401,
            "canonical-vid": 2401,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mMMv6qKvOBZX2fjZiB-An2QeS3Z1Te6RCaYWZe1ZWhp_PZLo_IUCFbtHxRf_Qxo7B9VdcIP1niV6AZIV0R-524AcojriuSYP17sRZLh6MVkUrbb90Gn80Tfznzci7nAdATeyzNX",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2401",
            "properties": {
                "firstname": {
                    "value": "tdlkjf"
                },
                "lastmodifieddate": {
                    "value": "1592571914163"
                },
                "createdate": {
                    "value": "1587046639836"
                },
                "lastname": {
                    "value": "ljkdfjd"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2401,
                    "saved-at-timestamp": 1587046639899,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "teste@teslj.com",
                            "timestamp": 1587046639836,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "35b28d70-acb1-4d88-b1b0-79e8ecff4c12",
                            "timestamp": 1587046639893
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592571914163,
            "vid": 2451,
            "canonical-vid": 2451,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mNFCUPkUyrWyBdy9vk2cyO6HteHzbc2E3DJLOoEcKJDAeK75A_sXEXwlzk_Sgjrjx7zUJnv632Q7PtG9xNAy5Qa0RgSXKbonrl_6aWyxEbRTuahU4HONS_UywYDMeSF-TybOL6v",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2451",
            "properties": {
                "firstname": {
                    "value": "dljfd"
                },
                "lastmodifieddate": {
                    "value": "1592571914163"
                },
                "createdate": {
                    "value": "1587046748743"
                },
                "lastname": {
                    "value": "ljdkfd"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2451,
                    "saved-at-timestamp": 1587046748774,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "teste@djklfj.com",
                            "timestamp": 1587046748743,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "bba49b54-4bbf-487b-bf4c-faebe1b1424c",
                            "timestamp": 1587046748770
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592571914128,
            "vid": 1751,
            "canonical-vid": 1751,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mPNpEgDv7B8csSxVdTPyRjfauhKSOzdERa3kU6SFv2Eyj5YBoNZJzKax7A7MF0UQ9o_chHhp6Fk-StBaquSGIXctYNzoe3uhknoOczO6xRJSQ0k9zPQNh0xgNTO-PLaSjaFqwGo",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/1751",
            "properties": {
                "firstname": {
                    "value": "lDjicgICM"
                },
                "lastmodifieddate": {
                    "value": "1592571914128"
                },
                "createdate": {
                    "value": "1586837436695"
                },
                "lastname": {
                    "value": "ucxHeLnDg"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 1751,
                    "saved-at-timestamp": 1586837437053,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "EMAIL",
                            "value": "qcuvzjrna@rngbhihwr.com",
                            "timestamp": 1586837436695,
                            "is-primary": true
                        },
                        {
                            "type": "LEAD_GUID",
                            "value": "01da6d2f-46ec-44ff-b0a9-a72c014e7965",
                            "timestamp": 1586837437049
                        }
                    ]
                }
            ],
            "merge-audits": []
        },
        {
            "addedAt": 1592571914062,
            "vid": 2551,
            "canonical-vid": 2551,
            "merged-vids": [],
            "portal-id": 6907359,
            "is-contact": true,
            "profile-token": "AO_T-mON8BmnqKfQjXSp6jb1WuuQuSWdjbrv79_tE-mD7AKT1M1QYzDUeGAQZmKUj8X3n4fDuWoBcdGZz0wECaoTOg88Thx9mg3vfufwCMsz5X8QSuRGOXrOZ9DjwzyzWdCd8pSFwwHe",
            "profile-url": "https://app.hubspot.com/contacts/6907359/contact/2551",
            "properties": {
                "firstname": {
                    "value": "t"
                },
                "lastmodifieddate": {
                    "value": "1592571914062"
                },
                "createdate": {
                    "value": "1587316271303"
                },
                "lastname": {
                    "value": "c"
                }
            },
            "form-submissions": [],
            "identity-profiles": [
                {
                    "vid": 2551,
                    "saved-at-timestamp": 1587394691329,
                    "deleted-changed-timestamp": 0,
                    "identities": [
                        {
                            "type": "LEAD_GUID",
                            "value": "ef6f4e1b-4cb6-440c-b476-1c1c653153f2",
                            "timestamp": 1587316271348
                        },
                        {
                            "type": "EMAIL",
                            "value": "updated@randomemail.com",
                            "timestamp": 1587394691253,
                            "is-primary": true
                        }
                    ]
                }
            ],
            "merge-audits": []
        }
    ],
    "has-more": true,
    "vid-offset": 2551,
    "time-offset": 1592571914062
}
0 Upvotes
WendyGoh
Solution
HubSpot Employee
HubSpot Employee

Get recently modified or created contacts bug - Indexing delay

SOLVE

Hey @tclark228,

 

Just an update here!

 

At the moment, the Get recently updated and created contacts endpoint reads from ElasticSearch to find contacts with a recent last-modified-date, and returns them in order of their last-modified-date (most recent first).

 

In your case, it looks like contact ID 3051's update was indexed into ElasticSearch a little bit delayed, just after your team had done the first query but before your team had repeated it the second time.

 

While we aim for a <5s index time from the update, the bulk updates may sometime take longer.

0 Upvotes
tclark228
Member

Get recently modified or created contacts bug - Indexing delay

SOLVE

Thanks Wendy. That makes sense. Appreciate the help

0 Upvotes
WendyGoh
HubSpot Employee
HubSpot Employee

Get recently modified or created contacts bug - Indexing delay

SOLVE

Hey @tclark228,

 

Thanks for sharing the detailed steps.

 

I'm now able to reproduce this issue and my current hypothesis is that because the timestamp is the same even to the seconds, that is why there's some change in the position. 

 

Having said that, I do notice that the miliseconds timestamp are all unique and so I'd expect the return response result to be the same. For this case, I'll have to check in with the internal team and I'll keep you posted here.

0 Upvotes
tclark228
Member

Get recently modified or created contacts bug - Indexing delay

SOLVE

Hi Wendy,

Thank you for your response. I've been able to reproduce it pretty often and the delays sometimes differ. 

I edited 10 contacts' state/region property in Hubspot by selecting 10 contacts then clicking edit from the main list/view all contacts page

I then made an API call to the get recently modified contactsAPI with query param count=10. On this request I get back 10 last modified contacts. Notice that the first one (the most recent) has a timestamp of 1593001672723

1st request response https://pastebin.com/NZMEkivK

 

The contact IDs are returned in the order of  
2751
2851
2801
2901
3401
2651
2401
2451
1751
2551

I make the same request a few moments later and get back the following 

Note that the first contact is still 2751 with the same timestamp as before - 1593001672723

2nd request response  https://pastebin.com/Ep61jfMi


However this time the order is 
2751
2851
2801
2901
3051
3401
2651
2401
2451
1751

3501 was added retroactively. 

The only way we know to solve for this essentially processes all contacts twice using this endpoint which uses a lot of our API requests and makes this process very inefficient. Our hope is that Hubspot can fix this issue so that contacts are not added to this list retroactively. 

The maximum delay we've seen is 22 minutes but it is hard to say that the delay won't ever be longer than that.

For more context we are trying to capture any contacts that are updated in the Hubspot system. The issue we have with webhooks is that we can't support all of our customers custom fields. We also looked at using the new V3 Search API but also found the same delay issues with that API in addition to some other weird behaviors. 

 

We greatly appreciate your help!

0 Upvotes