Enable A/B testing in workflow/automated emails

I was disappointed to see that I can't do an A/B test in an automated email:

https://knowledge.hubspot.com/articles/kcs_article/email/how-to-create-an-a-b-test-email

 

Marketo (a HubSpot competitor) allows this functionality.

Can you add this to the product sometime?

It's particularly helpful with a Welcome email sequence which is key to on-boarding new customers.

HubSpot updates
Feb 14, 2020

Hi all! Two quick updates for you on this:

 

Number 1: Progress is going well, we have our new A/B framework in good shape and at the moment we're focusing on the new in-app features needed to begin/end and analyse this new type of A/B test for workflow emails. This is a big update on what is currently available for batch emails, and will give you the ability to end an A/B test and start a new one on the same email without needing to rebuild the email over and over again. Getting very close now! 

 

Number 2: Moving forward this feature, the updates on it's release and all feedback will be handled by my colleague @jaleonard19. Jake is also a PM here working on our marketing email tools and has actually been the one spearheadding this feature for some time now so you'll be in great hands. 

 

Jake is currently working on our release plans and will be sharing a public beta signup form with you all as soon as it's ready. Thank you all again for your feedback and suggestions. Looking forward to seeing the impact this will have for you all.

Dec 19, 2019

Will do @petersherlock93

Dec 11, 2019

Will do @EJVFernyhough ! We'll be sharing a beta sign up form with everyone on this post when it's available so everyone will have the opportunity to sign up once it's open.

Dec 11, 2019

@AJernigan can do Smiley Happy

Nov 18, 2019

Hi @ridingforlife I can definitely share more. At the moment we're making good progress but factoring in the upcoming US and EU holiday season it's unlikely that we'll be hitting beta-ready in Q4 but safe to say that we'll have a beta in Q1 2020. As soon as we have an open beta we'll offer it up to everyone in this thread to give it a try. 

Nov 4, 2019

Hi all, just to keep everyone updated on our progress here, we've been making decent headway with the infrastructure planning and development roadmap for this feature and are making meaningful progress towards a beta.

 

There's still a lot of moving pieces here so we don't have a concrete date on when we'll be opening up a beta but I wanted to keep you all in the loop and confirm that we're definitely focused on this and are actively working on it.

Aug 14, 2019

Thanks @petersherlock93! Just a case of good ol' miscommunication Smiley Happy  The "on pause" here just means that we were still deep into the infrastructure and reliability work needed for this (and other) functionality. It was by no means deprioritised or taken off the roadmap, just needed to get the ordering right on the priorities to make sure we didn't put the cart in front of the horse, so to speak.

Aug 13, 2019

Hi @petersherlock93  (and all following along). I can confirm that this isn't the case, A/B testing of automated emails is still very much a priority for us.

 

For the past few months we have been focused primarily on work on the underlying infrastructure of email, to ensure speed, stability and reliability of the overall platform and user experience. We're nearing the end of this work right now which enables us to continue to build on top of the email tool and makes it possible for us to build more net-new functionality in a stable and scalable way, A/B testing of automated emails being a great example of that. 

 

While there hasn't been much news on this specific feature lately, it's something that is very high on our roadmap and now that we're seeing results from our infrastructure and performance work we're able to refocus our efforts on bringing these new features to the product. Rest assured that it's still something we're focused on. 

 

While we aren't yet at the stage of a beta for this feature, I'd recommend checking out these resources (one and two) from the HubSpot blog that provide interim solutions and as soon as we have a beta that's available for everyone I'll share a signup form for everyone here to try out. Thanks again for your feedback and for keeping the momentum going here. 

changed to: In Planning
Jan 29, 2019

Hi all, I wanted to share a quick status update with you on this. We've spent some time researching options for this feature and are making good progress. We'll soon be heading into a dedicated research phase and as we get close to a beta of this we'll open it up to everyone on this post to get early access. Please do keep your feedbac/upvotes coming, we really appreciate it.

Nov 16, 2018

Thank you all for sharing your feedback and use cases on this. The product team is currently thinking about how to solve this, and will post updates here. 

Apr 6, 2017

A/B testing an entire workflow to see what places could be improved to improve conversion rates.

213 Replies
New Contributor

Following this thread to access the beta sign up! 

New Member

I would also like to be included in the beta. Thanks!

Occasional Contributor

We would love to be included!

HubSpot Product Team
HubSpot Product Team

Hi Folks - 

As @Shay mentioned I will be taking over this thread. I wanted to post an update and ask a couple of questions to the group here. Here is the basic functionality we will be supporting in the Beta:

  • A simple 50-50 split between 2 variants 
  • Ability to start and end tests on the same automated email (without cloning and republishing). 
  • Ability to see a history of the tests run on that automated email. 
  • Manually choosing a winner based on open/ click data. 

When digging into this project we realized that AB testing for Automated emails differs from regular 1-off batch emails. The main difference is that Automated emails have the potential to last forever. Thus we needed to support the ability to start and end multiple tests on an Automated email. Tracking those changes and tests over time. Here are some questions I had for the group:

 

Q1: How do you all keep track of the different experiments and tests you have run today with Batch Email? Or with other tools (website pages, CTAs, etc.)?  

Q2: Are their other metrics that are important to picking a winning variant other than Open & Click information? What are they and how do you solve for that today?

Q3: What are the other important pieces of functionality that are missing from above? And which are the most important to you? (ex/ configurable percentages, multivariant, calculations to pick a winner, ability to have HS pick a winner, etc.) 

 

We are getting very close to having a Beta. Once we have it I will post a form in this thread in order for you all to join. We will then be quickly iterating and adding more advanced functionality.  

 

Occasional Contributor

@jaleonard19 This is exciting! 

 

Q1: We use excel to track tests and how they did. Would love something built-in.

Q2: We would love to narrow down click information to only track winning variation based on certain links clicked. For example, some emails may have a much higher click rate, but when you look at the clicks they are on the header, social media icons, and other things that mess up our results. We would love to have metrics just on the cta/links we care or are testing. 

Q3: We would love to test more than 2 emails. For instance, in Marketo, you can test multiple things by either having a randomizer (currently workaround is assigning to round-robin owner). Marketo also you can test up to 5 subject lines, which is super nice because often times we have more than two variables.

 

Thanks again for all your hard work!

 

Occasional Contributor

Hi Jake,

 

Thanks for keeping this moving - really excited to get this functionality rolling. Just to confirm - are you just testing beta-ing automated emails against each other or will you be able to beta test full workflows? Regarding your questions: 

 

Q1: How do you all keep track of the different experiments and tests you have run today with Batch Email? Or with other tools (website pages, CTAs, etc.)?   We tend to track testing as one-offs either through batching email or by running an automated test a certain way for a period of time and switching to a challenger layout and seeing if performance increases (obviously not a true test but often the best we can manage without true A-B automated email or workflow testing).

Q2: Are their other metrics that are important to picking a winning variant other than Open & Click information? What are they and how do you solve for that today? For us, while front-end metrics like opens and clicks are good initial barometers, we ultimately want to look at conversion information, which is often tied to actions on our site. So in a perfect world, we would be able to test two workflows against each other with the same goals and see which are performing better at converting on that workflow goal - this would allow a more holistic /accurate look at performance vs. just comparing open/click rates for a singular or automated email.

Q3: What are the other important pieces of functionality that are missing from above? And which are the most important to you? (ex/ configurable percentages, multivariant, calculations to pick a winner, ability to have HS pick a winner, etc.)  All of these would be great features, additionally see above re: full workflow testing with customizable 

Regular Contributor

@jaleonard19  Thanks for the update! In response to your questions...

 

Q1: How do you all keep track of the different experiments and tests you have run today with Batch Email? Or with other tools (website pages, CTAs, etc.)? I pull all of the metrics I need out of HubSpot and Google Analytics into excel. I find this to be the easiest way for me, and then the data is at that point is "static" and won't change when I revisit it. I can also easily combine results from multiple tests of the same thing (as I never only test something once), and see what the overall results are of what I was testing, in addition to each individual test.

 

Q2: Are their other metrics that are important to picking a winning variant other than Open & Click information? What are they and how do you solve for that today? Usually one of those metrics is the main KPI, as you can't control what happens after like on your website, but it can depend on what is being tested. In any case, we're definitely looking at conversions and revenue as well, which we get from Google Analytics as that is our main source across all channels when it comes to conversions and revenue.

 

Q3: What are the other important pieces of functionality that are missing from above? And which are the most important to you? (ex/ configurable percentages, multivariant, calculations to pick a winner, ability to have HS pick a winner, etc.) I would say all of those examples would be great to have, as well as the ability to test the workflow itself, so we're able to test other things than just an email in a workflow, like delays (i.e. test 3 days against 2 days) for example.

Occasional Contributor

Hi, just looking into email workflows A/B tetsing it would be helpfull to know when HS are looking to launch this tool its works in campaign monitor and Mail Chimp and rtaher than using other tools it would be good use wiithin the CRM tool. 

 

Thanks 

New Contributor

Upvoting this idea. I would love to see a solution to test emails on workflows.

For the moment I have to create seperate test lists to test the idea. This is ok for simple drip campaigns but can bee too complicated for more intuitive smart ones.... it's be great to have simplified process inplemented here. 

New Contributor

We really need this functionality, please!

Occasional Contributor

I can't wait to join the beta. We really need this functionality for our Lead Nurturing campaigns. 

New Contributor

I need it too

Occasional Contributor | Gold Partner
Occasional Contributor | Gold Partner

Following

Occasional Contributor

When can we get access to the beta?

Occasional Contributor

Would love to be included in this Beta, my team could really use this feature!

New Contributor

We'd love to be included in this beta as well!

 

Regular Visitor

Would love to be included in this Beta, we would like to do A/B testing with the titles of all the emails in our workflows to improve the open rate.

New Contributor

Hi HubSpot team! Just checking in here on status, I could have sworn you expected early to mid Q1 launch of this feature. Do we have an ETA?

 

Thanks!

Regular Visitor

Really excited about this feature and can't wait for the beta!

Occasional Contributor

Great idea - please keep me in the loop!