Workflow troubles: Monitoring many custom fields for a webhook post

[rbutler-nw]

Hi all,

We have an external system that we need our HubSport data to go to depening on certain conditions on the users profile.

The way we are setup is that we use Custom Fields for our newsletter opt ins across our many many brands. A brand can have multiple newsletters, so there are a lot of custom fields. In order to sync this information to our websites user ID system, we have a workflow setup that checks every custom field is "known" (so every time the value is updated) as an enrollment trigger (and re-enrollment trigger) so that HubSpot then posts the JSON over to our system to process the updates.

The issue we are having is that the amount of custom fields we are looking at is causing the Workflow page to lock up, and I am worried that we will soon hit a wall with adding more into the workflow. We have easily 100+ custom fields, and more will come as we expand our newsletter offerings further across our brands.

I am struggling to find a better way to do this. I attempted to use the HubSpot default propery of Last Modified Date, but unfortunately this field is updated on any minor change to the account (be it email activity, analycis activity etc) which would cause our system to get overloaded with the amount of users we have on our account. 

Any ideas on a good way for us to be able to monitor the updates to the custom fields without overloading the workflow enrollment triggers?

[rbutler-nw]

Hi all,

thanks for jumping on this so quickly.

To explain a little on how we use HubSpot:

We use the "Subscription" email types to classify email sends as Newsletters or Marketing. Due to the amount of brands we have on the single database and other issues with resubscriptions to those email subscription types, all of our Opt Ins are custom fields as subscription types isn't suitable to put all our brands on the HS unsub pages. We use our website and funnel our users to their profile pages to update opt ins. If they like they can globally unsub on the HS subscription pages, but typically users go to their profile for the respective brand and change the opt ins there.

Each custom field powers a list, and we have the same fields replicated on our ID system for our websites. 

As an example, a brand can have multiple newsletters, say news, sport, lifestyle etc. So each one of these newsletters is associated to a custom contact property for that brand only (we have over 60 brands on our account). We have a good few hundred custom properties across all of our brands on the system.

Updates on consent for those opt ins (simple yes/no) can come from the Forms API or an Embed, or can come directly via our "Middleware" when a user updates their opt ins on their website account - this method utilisises the HubSpot API and a private app to update the correct custom property for that record.

The workflow we have to sync changes back via the "Middleware" to our ID system ensures that user accounts match the marketing database, in the event they use a form submission (be it an API based one or embed) or if there are actions off the back of permission passes or other marketing activity within the HubSpot portal.

I have considered using Lists, but due to the scale of having to have 2 lists per opt in essentially (one to say are they opted in, one to say are they not to ensure the updates are captured for both yeses and no's) - scaling that up to all of our brands would likely put  me in the same situation where the workflow has a massive amount of conditions on the triggering and re-enrollement criteria, as well as biting into our List limits.

Since I have posted the above, I have had a thought of breaking the "monitoring" out into multiple smaller workflows just looking at a subset of brands with room for them to grow, and associate each of these smaller workflows with a custom "admin" property that updates every time a user qualifies. And then our original sync workflow could just monitor these workflow specific "admin" properties instead, which would be way fewer than all of the opt in fields. A working theory at the moment while I do some exploritory testing.

[CHawkins · Accepted solution]

I agree, I think multiple workflows might be your best avenue. 

It probably feels daunting to build and manage but I think this process is complicated so it needs a complicated solution.

@GeorgeBThomas told me recently "I love to think about workflows in a modular fashion. That way, when something (not if) breaks, I'm only fixing the axel, the muffler, or the starter, and not "buying" a brand-new car! - I have taught modular workflows and the idea of enrolling in the next workflow for years!"

I have been trying to implement that thought process in my own work.

[CHawkins]

Thanks for tagging me @DianaGomez!

@rbutler-nw - How are these custom fields added to the contact record?

My thought is if it is through a contact submitting from(s), you could use an enrollment trigger of "Recent Conversion Date" in the way you tried to use "Last Modified Date" already.

Alternatively, I am wondering if using a list for your enrollment would behave/scale more easily.  

[PamCotton]

Hello, @rbutler-nw thank you for posting in our Community and providing us with all these details.

I want to invite our top experts to this conversation  @Sdiddy407 @JDobson @rikkilear any recommendations for @rbutler-nw matter?

Thank you,

Pam

[DianaGomez]

Hi @rbutler-nw, 

Thank you for your inquiry. 

I'd like to involve some of our experts in the conversation to review whether they have any suggestions for you or if they have encountered a similar case in their experience and how they resolved it.

Hi @CHawkins, @Phil_Vallender, @Josh Do any of you happen to have any suggestions for @rbutler-nw ?

Thank you all for your contributions.

Best regards,

Diana