We have been working on cleaning up our duplicate contacts. We had a lot of internal structure changes and this completely fell to the wayside for probabaly about 2 years. A couple of months ago we had 2,000+ duplicates. We dedicated a lot of time and got the number down to 0. Less than a month later, we’re back up to 2,000. We decided to look into an automation tool, but it was costly and didn’t detect even half of the duplicates that HubSpot was showing.
We were able to build out a script internally to automate the merging of accounts with typos (like gmai.com rather than gmail.com). We were able to get it down to roughly 1,000 and have manually been cleaning up a couple hundred a day. The thing is, the next day it is right back up to around 1,000.
If I look at just the duplicates that were detected today, it is showing roughly 250. The majority of the contacts in the duplicate detection were created over a year ago . I am not sure if HubSpot is just taking a while to catch up and populate additional duplicates or if these are ones that were rejected previously and are coming up again. Either way, we cannot dedicate hours to contact cleanup each day. Any thoughts or suggestions are greatly appreciated!
Hi @MHulleman,
HubSpot’s duplicate detection is not in real time - you should see something like a “Last duplicate check” date time stamp in the tool, potentially along with a note like:
"Last calculation found more than … duplicates.
HubSpot displays up to … of the most likely duplicates, but you might have more. If you remove duplicates from this list, more will be displayed on the next calculation date."
This is expected behavior currently and there aren’t any settings to change it. If you’d like this behavior changed, you could request it from the product team in the ideas section of the community: https://community.hubspot.com/t5/HubSpot-Ideas/idb-p/HubSpot_Ideas
At this point, the strongest solutions for deduplication are still third-party integrations, like you mentioned, such as Dedupely, Insycle or Koalify installed (Disclaimer: the link is an affiliate link which makes me eligible for commission).
Personally, I recommend Koalify, linked above. It lets you set custom deduplication rules, primary rules and will store results in custom properties that you can filter for, use in reports or within workflows: https://koalify.io/
It’s a trade-off between the time you’re currently spending on manually merging duplicates on a regular basis and the subscription costs.
Best regards
In a lot of cases, the duplicate tool is also delayed, so HubSpot may still be surfacing older duplicate suggestions over time rather than all at once. I’ve seen that happen on large portals.
A few things that usually help long term:
Make email the primary unique identifier everywhere possible
Add validation for common domain typos before contact creation
Restrict manual contact creation where possible
Review integrations that may bypass duplicate checks
Create workflows to flag suspicious records immediately instead of waiting for duplicate detection