Identifying duplicates and marking contacts as primary/secondary

[karstenkoehler]

Hey everyone,

In a portal I'm overseeing, we're trying to identify duplicates of contacts who sign up for a software trial. We don't want to merge or delete these contacts, the goal is to keep both and label them accordingly (via custom properties) as primary contact and secondary contact.

For context, users sign up for multiple software trials, using different email addresses. Even if we let them know they shouldn't, they do. Still, we want to make sure our email marketing is on point and not redundant (not emailing multiple duplicates at the same time). The goal is to only email the primary duplicate, not any of the others. These need to be excluded from email marketing – but still be kept for reporting.

More details on the requirements:

• Contacts should be identified as a duplicate based on their first name, last name, company name OR first name, last name, phone number.
• The number of duplicates might be two but also higher.
• One of these contact records contains more behavioral data about the software trial than the other duplicates. This contact should be marked as "Primary duplicate" in a custom property. All other duplicates would be a "Secondary duplicate".
• In case there is no legal basis for communicating with the primary contact, a second custom property needs to be maintained, "Duplicate to be marketed to". This might be the primary or one of the secondary contacts. Whichever contact has an opt-in for a subscription type and the most behavioral data.

So far, we're thinking that this can be achieved with Operations Hub Professional and custom code actions. (There are other use cases we have for Operations Hub.) Contacts would be enrolled, the script would execute actions as per the requirements above, based on the CRM API endpoint.

Does this approach make sense?

Is there an easier way to achieve this?

Are we missing something obvious?

(@dennisedson, if this is something I could achieve with basic Javascript knowledge and resources to skill up, let me know. In that case, could you point me in the right direction?)

Additional notes:

• If you feel like this is a job for you, please let me know. We're looking for a freelancer or a agency to support us here. The company is based in Germany.
• We've evaluated Insycle but the pricing is prohibitive with our contact volume. Also, it doesn't meet all requirements.
• If there's a way to achieve this without Operations Hub Professional, we're also open to it – although there are other use cases we have for Operations Hub Professional.

Thanks in advance!

[dennisedson · Accepted solution]

@karstenkoehler , this is not what I meant when I said I wanted to see you over here 🤣

This dedupe custom workflow is a good starting point. 

You would need to add a couple more variables in addition to "phone"

const DEDUPE_PROPERTY2 = 'firstname';
const DEDUPE_PROPERTY3 = 'lastname';

after line 21, you woudl include your two new props

let dedupePropValue2 = contactResult.body.properties[DEDUPE_PROPERTY2];
let dedupePropValue3 = contactResult.body.properties[DEDUPE_PROPERTY3];

Then update the search to include them

.doSearch({
          filterGroups: [{
            filters: [{
              propertyName: DEDUPE_PROPERTY,
              operator: 'EQ',
              value: dedupePropValue
            },
            {
              propertyName: DEDUPE_PROPERTY2,
              operator: 'EQ',
              value: dedupePropValue2
            },
            {
              propertyName: DEDUPE_PROPERTY3,
              operator: 'EQ',
              value: dedupePropValue3
            }
          }]
        })

after this search occurs and nothing matches, you should get a console log saying this: No matching contact, nothing to merge .  You can update that to whatever floats your boat.

------

What indicates this:

One of these contact records contains more behavioral data about the software trial than the other duplicates.

This will determine how you work with the next step. 

-------

On line 42, we should probably change the length to 2 at a minimum.  From what I am gathering, the record for both already exist so we should expect there to be at least 2, but this could else statement might be removed altogether in the end.

For now, we can pretend that there was a match and from whatever was decided from the missing piece above, we can action the dupe.

For that, let's focus on the request starting at line 49 and ending at 56

Instead of merging the contacts, you want to assign a value to a custom property.  Lets call that prop "dupe"

hubspotClient
  .apiRequest({
    method: 'PATCH',
    path: `/crm/v3/objects/contacts/IDOFSECONDARY`,
    body: {
     "properties": {
        "dupe": "duplicate"
     }
    }
  })

This is a very very rough draft of how I would start to think about it.  @Teun and @Bryantworks  are both much much smarter people than I am and will be able to get you over the finish line.  You will need to clarify how you determine which is primary, though.

[Bryantworks · Accepted solution]

Thanks for the tag @dennisedson!  @karstenkoehler, have you tried Insycle?  Doing this in a custom code workflow is a little iffy to me at the moment, but Insycle has complex deduping on the fields you mentioned as well as master rules, defining which record wins based on other criteria (more engagement vs none, etc.).  

Now, dedupe.ly is cheaper, but Insycle you can run recipes via workflow meaning when a contact comes in, run the duplicate check in seconds before your email marketing starts to hit.

Maybe that would work?

[Teun · Accepted solution]

Hi @karstenkoehler and @dennisedson ,

The Search API is indeed the best first step in this custom workflow. The hard part is probably determining which contact should be the primary contact.

You could try to achieve this by getting additional information in your custom-coded action, however, I would probably create some custom scoring mechanism where you use the events and Operations Hub to increase the score (could be a number property) of each contact based on behavioral events (much like lead scoring,  but in this case specifically for the software trials).

So let's say a contact visited a few pages that are about the software trial, you can use a workflow to increase the custom score property by a certain number. 

That way, you can use the custom-coded action to simply compare the custom property score of each contact to determine which one should be the primary duplicate. If you have that if statement set up, you can include the legal basis as well to determine the right properties for each contact.

[karstenkoehler]

Hi @Teun,

Thank you for your reply, that's a great way of going about this, hadn't thought about it that way yet.

Cheers!

[Bryantworks · Accepted solution]

Thanks for the tag @dennisedson!  @karstenkoehler, have you tried Insycle?  Doing this in a custom code workflow is a little iffy to me at the moment, but Insycle has complex deduping on the fields you mentioned as well as master rules, defining which record wins based on other criteria (more engagement vs none, etc.).  

Now, dedupe.ly is cheaper, but Insycle you can run recipes via workflow meaning when a contact comes in, run the duplicate check in seconds before your email marketing starts to hit.

Maybe that would work?

[karstenkoehler]

Hi @Bryantworks,

With a few hundred thousands of contacts, Insycle becomes too expensive, unfortunately. From a first glance it's also not as flexible as it would have to be.

Thank you still!

[dennisedson · Accepted solution]

@karstenkoehler , this is not what I meant when I said I wanted to see you over here 🤣

This dedupe custom workflow is a good starting point. 

You would need to add a couple more variables in addition to "phone"

const DEDUPE_PROPERTY2 = 'firstname';
const DEDUPE_PROPERTY3 = 'lastname';

after line 21, you woudl include your two new props

let dedupePropValue2 = contactResult.body.properties[DEDUPE_PROPERTY2];
let dedupePropValue3 = contactResult.body.properties[DEDUPE_PROPERTY3];

Then update the search to include them

.doSearch({
          filterGroups: [{
            filters: [{
              propertyName: DEDUPE_PROPERTY,
              operator: 'EQ',
              value: dedupePropValue
            },
            {
              propertyName: DEDUPE_PROPERTY2,
              operator: 'EQ',
              value: dedupePropValue2
            },
            {
              propertyName: DEDUPE_PROPERTY3,
              operator: 'EQ',
              value: dedupePropValue3
            }
          }]
        })

after this search occurs and nothing matches, you should get a console log saying this: No matching contact, nothing to merge .  You can update that to whatever floats your boat.

------

What indicates this:

One of these contact records contains more behavioral data about the software trial than the other duplicates.

This will determine how you work with the next step. 

-------

On line 42, we should probably change the length to 2 at a minimum.  From what I am gathering, the record for both already exist so we should expect there to be at least 2, but this could else statement might be removed altogether in the end.

For now, we can pretend that there was a match and from whatever was decided from the missing piece above, we can action the dupe.

For that, let's focus on the request starting at line 49 and ending at 56

Instead of merging the contacts, you want to assign a value to a custom property.  Lets call that prop "dupe"

hubspotClient
  .apiRequest({
    method: 'PATCH',
    path: `/crm/v3/objects/contacts/IDOFSECONDARY`,
    body: {
     "properties": {
        "dupe": "duplicate"
     }
    }
  })

This is a very very rough draft of how I would start to think about it.  @Teun and @Bryantworks  are both much much smarter people than I am and will be able to get you over the finish line.  You will need to clarify how you determine which is primary, though.

[karstenkoehler]

Thanks @dennisedson, this is great!

---

@dennisedson wrote:

One of these contact records contains more behavioral data about the software trial than the other duplicates.

---

In most cases this will be the contact which was created first – which should make this easy. Other behavioral data are feed into HubSpot via contact properties and behavioral events, so I assume we should be able to reference those.

Thanks!