Enable A/B testing in workflow/automated emails

I was disappointed to see that I can't do an A/B test in an automated email:

https://knowledge.hubspot.com/articles/kcs_article/email/how-to-create-an-a-b-test-email

 

Marketo (a HubSpot competitor) allows this functionality.

Can you add this to the product sometime?

It's particularly helpful with a Welcome email sequence which is key to on-boarding new customers.

HubSpot updates
Status updated to: In Beta
a week ago

Hi Everyone,

 

Thank you for your pateince here. I am ecstatic to be updating this post to "In Beta" today. We rolled this feature out to a list of Beta users from this morning. If you want to be added to it please reach out to support and they can request it through the proper channels. 

We know this has been a very popular request and the team have been working so hard to deliver this as fast as possible. We hope to be rolling it out to a wider audience in coming weeks.

Best,

 

Shane

Status updated to: Being Reviewed
Dec 2, 2020

Status updated to: Idea Submitted
Nov 23, 2020

Hi All! Again thank you for your patience on this feature. Again we are almost there with some final checks and tests. We hope to have this released to Beta next week. I will update this post when it is live. Please do reach out to your CSM if you want to be part of the Beta - By this I mean send an email to support and they can add you to the Beta list.

 

We are really excited to get this out after months of hard work by the team here.

 

Thank you!


Best,

 

Shane Janssens

Nov 13, 2020

Hi All, 

I just wanted to update the recent questions @petersherlock93 , @jmcguire20 , @stuartborders , @JPD123 

 

We are in final testing here. expecting to release to beta group in late Nov. If you want to get siogned up for Beta please reach out to your CSM and they can add you to list.

 

I commented with updates but they seem to be lost in thread. Sorry about confusion.

 

Best,

 

Shane

Sep 30, 2020

Hi all,

 

Quick update here. Again thank you for your patience and feedback here. We are currently moving to testing this feature. It will take a bit of time as it is a big change on the email infastructure. I will update here as soon as it is in Beta. I wont give an exact timing as it depends on what we find in testing. Anyone who has signed up will be in the first release group, if you have not asked support for this please reach out and they can add you to the beta list.

 

Thank you, it is very close now.

 

Best,

 

Shane

 

Status updated to: Being Reviewed
Sep 9, 2020

Hi Everyone,

 

Thank you so much for all your interest and comments in this forum. I am the Product Manager on Marketing Email and I wanted to jump in here as I have been closest to this project for past few months. We are so close, we had hit a couple of small bumps that set us back with the complexity of this build. We are in the final stretch and we are talking about delivering this in September. We have had 2 teams involved in building this and we are so excited to release the Beta as soon as we can. I will update this post as soon as it is ready for beta.

Thank you for your patience and continued interest in this project.

 

Best,

 

Shane

Feb 14, 2020

Hi all! Two quick updates for you on this:

 

Number 1: Progress is going well, we have our new A/B framework in good shape and at the moment we're focusing on the new in-app features needed to begin/end and analyse this new type of A/B test for workflow emails. This is a big update on what is currently available for batch emails, and will give you the ability to end an A/B test and start a new one on the same email without needing to rebuild the email over and over again. Getting very close now! 

 

Number 2: Moving forward this feature, the updates on it's release and all feedback will be handled by my colleague @jaleonard19. Jake is also a PM here working on our marketing email tools and has actually been the one spearheadding this feature for some time now so you'll be in great hands. 

 

Jake is currently working on our release plans and will be sharing a public beta signup form with you all as soon as it's ready. Thank you all again for your feedback and suggestions. Looking forward to seeing the impact this will have for you all.

Dec 19, 2019

Will do @petersherlock93

Dec 11, 2019

Will do @EJVFernyhough ! We'll be sharing a beta sign up form with everyone on this post when it's available so everyone will have the opportunity to sign up once it's open.

Dec 11, 2019

@AJernigan can do Smiley Happy

Nov 18, 2019

Hi @ridingforlife I can definitely share more. At the moment we're making good progress but factoring in the upcoming US and EU holiday season it's unlikely that we'll be hitting beta-ready in Q4 but safe to say that we'll have a beta in Q1 2020. As soon as we have an open beta we'll offer it up to everyone in this thread to give it a try. 

Nov 4, 2019

Hi all, just to keep everyone updated on our progress here, we've been making decent headway with the infrastructure planning and development roadmap for this feature and are making meaningful progress towards a beta.

 

There's still a lot of moving pieces here so we don't have a concrete date on when we'll be opening up a beta but I wanted to keep you all in the loop and confirm that we're definitely focused on this and are actively working on it.

Aug 14, 2019

Thanks @petersherlock93! Just a case of good ol' miscommunication 🙂  The "on pause" here just means that we were still deep into the infrastructure and reliability work needed for this (and other) functionality. It was by no means deprioritised or taken off the roadmap, just needed to get the ordering right on the priorities to make sure we didn't put the cart in front of the horse, so to speak.

Aug 13, 2019

Hi @petersherlock93  (and all following along). I can confirm that this isn't the case, A/B testing of automated emails is still very much a priority for us.

 

For the past few months we have been focused primarily on work on the underlying infrastructure of email, to ensure speed, stability and reliability of the overall platform and user experience. We're nearing the end of this work right now which enables us to continue to build on top of the email tool and makes it possible for us to build more net-new functionality in a stable and scalable way, A/B testing of automated emails being a great example of that. 

 

While there hasn't been much news on this specific feature lately, it's something that is very high on our roadmap and now that we're seeing results from our infrastructure and performance work we're able to refocus our efforts on bringing these new features to the product. Rest assured that it's still something we're focused on. 

 

While we aren't yet at the stage of a beta for this feature, I'd recommend checking out these resources (one and two) from the HubSpot blog that provide interim solutions and as soon as we have a beta that's available for everyone I'll share a signup form for everyone here to try out. Thanks again for your feedback and for keeping the momentum going here. 

Status updated to: In Planning
Jan 29, 2019

Hi all, I wanted to share a quick status update with you on this. We've spent some time researching options for this feature and are making good progress. We'll soon be heading into a dedicated research phase and as we get close to a beta of this we'll open it up to everyone on this post to get early access. Please do keep your feedbac/upvotes coming, we really appreciate it.

Nov 16, 2018

Thank you all for sharing your feedback and use cases on this. The product team is currently thinking about how to solve this, and will post updates here. 

Apr 6, 2017

A/B testing an entire workflow to see what places could be improved to improve conversion rates.

258 Replies
New Contributor

@bobaloobadoo you can create additional user property (i called mine `test_group`) and set that field in your workflow rotation to keep `Contact Owner` usable in CRM.

HubSpot Employee

This would be a really great feature! +1 This Idea

Occasional Contributor

Totally +1 this idea. 

Your logic behind not being able to AB test on automated emails is completely flawed - following the same logic, Google should not also allow this kind of tests on their ads!

It's a complete fail on your part, Hubspot, that this functionality is unavailable.

Regular Contributor

Have to agree this reply makes no sense. 

 

You enable a/b testing on landing pages, but people visit landing pages at different times of day. 

 

If it makes sense to A/B test ad content or landing pages which deliver at different times of day to individual people, it 100% makes sense to A/B test emails in a flow.

 

It's about improving the flow, and if you are sending a large volume of emails the time of day issue should not skew the data. 

New Contributor

This would be a very beneficial addition to HubSpot to be able to A/B Test in a workflow. We have several onboarding email workflows for new customers and being able to test different emails for new users would be helpful.

Occasional Contributor

Adding my voice fo support to this thread in the hopes this becomes a feature in the near future. The inability to A/B split an automated email seems like a massive feature gap to me. the value of Hubspot to me woud be dramatically increased if A/B slitting automated emails was a feature. 

New Contributor | Platinum Partner

+ 1 we really need this feature for us and our clients 

 

I follow this article : https://blog.hubspot.com/customers/how-to-a/b-test-workflow-emails-using-the-lead-rotator-or-zapier but this is complicated to set up 

Occasional Contributor

Agreed - this makes no sense and not having A/B test means there is no way at all of testing a change (i.e. a text link or a button link). There may be a point regarding different times of day, but without any form of testing there is nothing at all to test against an we are in the dark of any changes made.

 

Going to try out the following: https://blog.hubspot.com/customers/how-to-a/b-test-workflow-emails-using-the-lead-rotator-or-zapier as some form of A/B testing is paramount to us evolving email 

Occasional Contributor

this is non-sensical. 

this needs to be implemented.

Status updated to: In Planning
HubSpot Product Team

Hi all, I wanted to share a quick status update with you on this. We've spent some time researching options for this feature and are making good progress. We'll soon be heading into a dedicated research phase and as we get close to a beta of this we'll open it up to everyone on this post to get early access. Please do keep your feedbac/upvotes coming, we really appreciate it.

Occasional Contributor

Thank you Shay, that's lovely to hear!

Occasional Contributor

This is the single largest feature gap of HubSpot. We could literally learn so much, so much faster if this was a feature. The day this gets released I will sing delighted praises to the name of Dharmesh Shah. 

 

Jorgen

Occasional Contributor

I think this feature is really missing. I would be absolutely necessary for us.

Occasional Contributor

This is a MUST for us. Please prioritize it 

New Contributor

+1 for this Idea... its totally necessary!!

New Member

+1 - Seems so basic.  I saw another tool that handled it beautifully.  Hubspot is a ltech eader and I expected to find it  in the tool.

New Contributor | Platinum Partner

I hate that HubSpot doesn't offer this feature as well. 

 

I am constantly working to optimize my workflows and the best way I have found to do it is super manual but it has been working. 

 

I write down the day I implemented an email change and monitor it on a monthly basis until it reaches statistical significance using this calculator: https://www.verticalmeasures.com/resources/ppc/statistical-significance-calculator/ 

 

Once the test has reached at least 95% statistical significance I conclude if the test won or was unsuccessful before determining if I will keep this change in place. 

 

It's not the most efficient but it has been working for me! 

Occasional Contributor

This is great to see in the planning and research phases, it's a huge need for our email marketing campaigns. 

Occasional Contributor

When will Hubspot be implementing this feature? If outreach.io, marketo, etc. can do it, then surely Hubspot can!

-Angela

New Contributor

I've just set a workflow up using an IF/THEN split and duplicated automated emails with different subject lines (or whatever other element you wish to test).

 

IF "email" starts with "a" through to "m", they get Version 1. If it's "n" through to "z", they get Version 2.

 

It's probably not the best A/B testing methodology, but it should work.