Enable A/B testing in workflow/automated emails

katharine

I was disappointed to see that I can't do an A/B test in an automated email:

https://knowledge.hubspot.com/articles/kcs_article/email/how-to-create-an-a-b-test-email

 

Marketo (a HubSpot competitor) allows this functionality.

Can you add this to the product sometime?

It's particularly helpful with a Welcome email sequence which is key to on-boarding new customers.

HubSpot updates
Jan 15, 2021

Hey folks!

 

The ability to A/B test to automated emails is now in public beta. This enables you to try new tactics and optimize your email sending. To A/B test an email, open the email editor and click "run a test" in the top left. 

Status updated to: In Beta
Jan 13, 2021

Hi Everyone,

 

Thank you for your pateince here. I am ecstatic to be updating this post to "In Beta" today. We rolled this feature out to a list of Beta users from this morning. If you want to be added to it please reach out to support and they can request it through the proper channels. 

We know this has been a very popular request and the team have been working so hard to deliver this as fast as possible. We hope to be rolling it out to a wider audience in coming weeks.

Best,

 

Shane

Status updated to: Being Reviewed
Dec 2, 2020

Status updated to: Idea Submitted
Nov 23, 2020

Hi All! Again thank you for your patience on this feature. Again we are almost there with some final checks and tests. We hope to have this released to Beta next week. I will update this post when it is live. Please do reach out to your CSM if you want to be part of the Beta - By this I mean send an email to support and they can add you to the Beta list.

 

We are really excited to get this out after months of hard work by the team here.

 

Thank you!


Best,

 

Shane Janssens

Nov 13, 2020

Hi All, 

I just wanted to update the recent questions @petersherlock93 , @jmcguire20 , @stuartborders , @JPD123 

 

We are in final testing here. expecting to release to beta group in late Nov. If you want to get siogned up for Beta please reach out to your CSM and they can add you to list.

 

I commented with updates but they seem to be lost in thread. Sorry about confusion.

 

Best,

 

Shane

Sep 30, 2020

Hi all,

 

Quick update here. Again thank you for your patience and feedback here. We are currently moving to testing this feature. It will take a bit of time as it is a big change on the email infastructure. I will update here as soon as it is in Beta. I wont give an exact timing as it depends on what we find in testing. Anyone who has signed up will be in the first release group, if you have not asked support for this please reach out and they can add you to the beta list.

 

Thank you, it is very close now.

 

Best,

 

Shane

 

Status updated to: Being Reviewed
Sep 9, 2020

Hi Everyone,

 

Thank you so much for all your interest and comments in this forum. I am the Product Manager on Marketing Email and I wanted to jump in here as I have been closest to this project for past few months. We are so close, we had hit a couple of small bumps that set us back with the complexity of this build. We are in the final stretch and we are talking about delivering this in September. We have had 2 teams involved in building this and we are so excited to release the Beta as soon as we can. I will update this post as soon as it is ready for beta.

Thank you for your patience and continued interest in this project.

 

Best,

 

Shane

Jul 24, 2020

Hi folks - Jumping in here again from the HubSpot Product team. I want to apologize for how long this has taken. This work is in progress and is a high priority for the team. Unfortunately, we have hit a number of snags during the development of this, which is slowing down the release. Hopefully in another month or so we should have clarity on timing. I'll post here as soon as we have more details to share. In the meantime please feel free to keep commenting, and upvoting the post.

Apr 20, 2020

@btuttle279 Absolutely - just sent over a message. We can jump on a call and walk through what this will all look like! 

Apr 20, 2020

Hi folks - Just posting a quick update on this thread. We are still focused on this project. Waiting on a couple of the backend changes to wrap up before we move into a Beta. Unfortunately, the work took a bit longer than originally anticipated. We are targeting getting the feature into a Beta this quarter (hopefully sometime in late May/ early June). As we roll out the Beta I will announce that in this forum, and post a form for you all to fill out to join. So hang tight for that!


We are very excited to get this one rolled out to you all and to start collecting feedback on it.

Feb 26, 2020

Hi Folks - 

As @Shay mentioned I will be taking over this thread. I wanted to post an update and ask a couple of questions to the group here. Here is the basic functionality we will be supporting in the Beta:

  • A simple 50-50 split between 2 variants 
  • Ability to start and end tests on the same automated email (without cloning and republishing). 
  • Ability to see a history of the tests run on that automated email. 
  • Manually choosing a winner based on open/ click data. 

When digging into this project we realized that AB testing for Automated emails differs from regular 1-off batch emails. The main difference is that Automated emails have the potential to last forever. Thus we needed to support the ability to start and end multiple tests on an Automated email. Tracking those changes and tests over time. Here are some questions I had for the group:

 

Q1: How do you all keep track of the different experiments and tests you have run today with Batch Email? Or with other tools (website pages, CTAs, etc.)?  

Q2: Are their other metrics that are important to picking a winning variant other than Open & Click information? What are they and how do you solve for that today?

Q3: What are the other important pieces of functionality that are missing from above? And which are the most important to you? (ex/ configurable percentages, multivariant, calculations to pick a winner, ability to have HS pick a winner, etc.) 

 

We are getting very close to having a Beta. Once we have it I will post a form in this thread in order for you all to join. We will then be quickly iterating and adding more advanced functionality.  

 

Feb 14, 2020

Hi all! Two quick updates for you on this:

 

Number 1: Progress is going well, we have our new A/B framework in good shape and at the moment we're focusing on the new in-app features needed to begin/end and analyse this new type of A/B test for workflow emails. This is a big update on what is currently available for batch emails, and will give you the ability to end an A/B test and start a new one on the same email without needing to rebuild the email over and over again. Getting very close now! 

 

Number 2: Moving forward this feature, the updates on it's release and all feedback will be handled by my colleague @jaleonard19. Jake is also a PM here working on our marketing email tools and has actually been the one spearheadding this feature for some time now so you'll be in great hands. 

 

Jake is currently working on our release plans and will be sharing a public beta signup form with you all as soon as it's ready. Thank you all again for your feedback and suggestions. Looking forward to seeing the impact this will have for you all.

Dec 19, 2019

Will do @petersherlock93

Dec 11, 2019

Will do @EJVFernyhough ! We'll be sharing a beta sign up form with everyone on this post when it's available so everyone will have the opportunity to sign up once it's open.

Dec 11, 2019

@AJernigan can do Smiley Happy

Nov 18, 2019

Hi @ridingforlife I can definitely share more. At the moment we're making good progress but factoring in the upcoming US and EU holiday season it's unlikely that we'll be hitting beta-ready in Q4 but safe to say that we'll have a beta in Q1 2020. As soon as we have an open beta we'll offer it up to everyone in this thread to give it a try. 

Nov 4, 2019

Hi all, just to keep everyone updated on our progress here, we've been making decent headway with the infrastructure planning and development roadmap for this feature and are making meaningful progress towards a beta.

 

There's still a lot of moving pieces here so we don't have a concrete date on when we'll be opening up a beta but I wanted to keep you all in the loop and confirm that we're definitely focused on this and are actively working on it.

Aug 14, 2019

Thanks @petersherlock93! Just a case of good ol' miscommunication 🙂  The "on pause" here just means that we were still deep into the infrastructure and reliability work needed for this (and other) functionality. It was by no means deprioritised or taken off the roadmap, just needed to get the ordering right on the priorities to make sure we didn't put the cart in front of the horse, so to speak.

Aug 13, 2019

Hi @petersherlock93  (and all following along). I can confirm that this isn't the case, A/B testing of automated emails is still very much a priority for us.

 

For the past few months we have been focused primarily on work on the underlying infrastructure of email, to ensure speed, stability and reliability of the overall platform and user experience. We're nearing the end of this work right now which enables us to continue to build on top of the email tool and makes it possible for us to build more net-new functionality in a stable and scalable way, A/B testing of automated emails being a great example of that. 

 

While there hasn't been much news on this specific feature lately, it's something that is very high on our roadmap and now that we're seeing results from our infrastructure and performance work we're able to refocus our efforts on bringing these new features to the product. Rest assured that it's still something we're focused on. 

 

While we aren't yet at the stage of a beta for this feature, I'd recommend checking out these resources (one and two) from the HubSpot blog that provide interim solutions and as soon as we have a beta that's available for everyone I'll share a signup form for everyone here to try out. Thanks again for your feedback and for keeping the momentum going here. 

Status updated to: In Planning
Jan 29, 2019

Hi all, I wanted to share a quick status update with you on this. We've spent some time researching options for this feature and are making good progress. We'll soon be heading into a dedicated research phase and as we get close to a beta of this we'll open it up to everyone on this post to get early access. Please do keep your feedbac/upvotes coming, we really appreciate it.

Nov 16, 2018

Thank you all for sharing your feedback and use cases on this. The product team is currently thinking about how to solve this, and will post updates here. 

Apr 6, 2017

A/B testing an entire workflow to see what places could be improved to improve conversion rates.

282 Replies
sarahcocos
Member

@ShaneJ 

Wanted to share some feedback on our experience with the rollout. First, SO excited this is finally live for testing and using. 
One thing I'm noticing that seems like a product oversight is that you can't search and find these automated A/B emails within 'marketing email activity' for workflow logic. 

Here's our use-case scenario: We have if/then branches for engagement (ie if they opened or clicked x/y/z email, then move them to a hot leads branch and if they're not opening/clicking, move them to the cold leads branch). So, when we use A/B Automated emails, i can't find them as options to use for these open/click branches to get info on their email activities. 

Is there a reason these aren't showing up in workflows as options? Wanted to flag this, as it's a large part of our automation and lead segmentation process. 

Thanks for any info and updates you can provide here. 

-Sarah 

RebeccaS
Participant

Hi Joe. I just created an email for automation but I don't see where I can enable the split test, can you provide a screenshot please? 

JoeMayall
HubSpot Product Team

Hi @RebeccaS ! 

 

Here you go! 

2.png3.png

RebeccaS
Participant

Hi @JoeMayall 

I don't have that, we use plain html emails, is that the issue?

 

RebeccaS_0-1612508118247.png

 

ridingforlife
Top Contributor

I don't see the A/B testing option available either, no matter what kind of automated email template is used.

 

Also to confirm, this allows for A/B testing of automated email content, but not workflow functionality (ex. testing delays between emails)?

RebeccaS
Participant

@ridingforlife 

 

I emailed HubSpot support and they gave me access (you have to request it). I am now able to see the option, but only for new emails I create (can't apply it to any existing).  Yes you can test the content/subject lines etc and view the results in the email dash, and choose a winner. As far as I know, it's not about testing workflow functionality. Hope this helps 🙂

sarahcocos
Member

@ShaneJ  and other Beta Users:
Just an FYI, the product team said the issue I reported earlier (not being able to search/select these A/B auto emails in the marketing emails logic in a workflow) has been fixed, but I'm running into these two other issues and curious if anyone else is seeing the same issues? 

 

1. When creating an if/then logic branch in a workflow to show anyone who's clicked or opened these A/B emails, I can see and select them, but then when you save that logic branch in the workflow, it throws an error on those emails that says 'This criteria couldn't be found. It may have been deleted.' so you can't actually use any marketing email activity data in the workflow with these emails. 

2. Contact records are not being updated with these A/B Automated marketing emails. I'm not seeing date stamps for these two fields: 'marketing emails delivered' or 'last marketing email sent'  when using these new beta emails.  

danielrnorris
Participant | Gold Partner

Excited to see this is in Beta, such an important feature for my clients.

SamJigSpace
Member

I would like to gain access to this beta program too. 

 

Will email my AM, but if someone else at Hubspot can activate it too that would be great.

 

Thanks!

Khalvin
Member

Hi, I would like to be enrolled into this beta program, thank you. 

ridingforlife
Top Contributor

I noticed documentation for A/B testing emails in workflows notes that an A/B email can only be used in 1 workflow:
https://knowledge.hubspot.com/workflows/automate-ab-emails-with-workflows

 

workflow-ab-test-email.jpg

 

 

 

 

Is there anything in the roadmap to allow an A/B email to be used across multiple workflows?

 

EDIT: I have created an idea on this here: https://community.hubspot.com/t5/HubSpot-Ideas/Use-A-B-emails-across-multiple-workflows/idi-p/421224

joelfed
Member

@JoeMayall - Happy to see this is in Beta. However, with an Automated email template open, I'm not seeing any option to "Run a Test" in either the upper left of the email editor or anywhere else on the page. Can you provide some further direction? It does show when I create a Regular email, but of course that's not what this thread is about.

 

Thanks,

 

Screen Shot 2021-03-25 at 2.15.55 PM.png

spogue
Top Contributor
@joelfed I had a similar experience. My issue was that I created a new
email from a clone. Not good enough apparently. You have to create a new
automated email from scratch and then copy in the new information. Hope
that helps!

--
Steve Pogue
*Marketing Operations Manager*
*Workzone*
p: 484.263.2461 <+14842632461>
e: steve@workzone.com
w: www.workzone.com

ridingforlife
Top Contributor

@joelfed  Have you reached out to support to request your portal be added to the beta?

joelfed
Member

@ridingforlife - No. An earlier message from the HubSpot team had indicated that it was in Public Beta, so I thought that meant available for all to use. But I'll be sure to request it.

jmacandrews
Participant

Can we be added to this beta?

fiachra_duffy
HubSpot Employee

@jmacandrews reach out ot HubSpot Support or your Customer Success Manager to be enrolled in the beta 🙂 

btuttle279
Participant

Cant delete comments, so just ignore this, messed up.

Drishtii_
Participant

Hi, 

I know that this feature is in Beta and I have been using this. 

 

While I love this, I do have a quick question here: 

Do these emails that go out as A/B tests via workflows have the source tracking right? And by that, I mean the usual UTM values that Hubspot attaches to the links on the emails. 

 

I ask this because I noticed that my form fills did not capture the UTM source, campaign, and medium values. I did check up on the landing page and the way it is set up. I tested the form out and it seems to be working fine. 

 

Am I missing something here? 
@JoeMayall 

APlocinik
Member

@JoeMayall the A/B functionality for automated emails is great, however I was very disappointed to just learn that I cannot use it across multiple workflows. We have separate nurture workflows for US, EMEA and APAC and do use some emails across all three. Is there any plan to make this functionality allowable across multiple workflwos once this moves beyond beta?