How to setup A/B testing in automated emails

SOLVE
New Contributor

I've built out a free trial workflow for new users who sign up for our community and would like to to do some A/B testing of the emails ( starting with a simple subject line test). 

 

I've noticed that once I have activated my workflow I am not able to add an A/B test in the email builder. Is there a way to add this in without pausing the workflow? 

 

Thanks,

 

Sandy 

1 Accepted solution

Accepted Solutions
Academy Team

Hello @Sandy

 

Currently, you won't be able to create an A/B test with an email that is saved for automated workflows. Make sure your email is set to send at a specific date/time or immediately (under the Sending tab in your email). 

 

The reason behind this is an automated email will continue to send over time at different intervals depending on when someone gets enrolled in the workflow. I would suggest if you wish to test some variables of your email you run an A/B Test on a regular single-send email and then depending on the results could implement that functionality into your automated email. 

 

Best, 

Courtney Sembler 

Inbound Professor 

35 Replies 35
Academy Team

Hello @Sandy

 

Currently, you won't be able to create an A/B test with an email that is saved for automated workflows. Make sure your email is set to send at a specific date/time or immediately (under the Sending tab in your email). 

 

The reason behind this is an automated email will continue to send over time at different intervals depending on when someone gets enrolled in the workflow. I would suggest if you wish to test some variables of your email you run an A/B Test on a regular single-send email and then depending on the results could implement that functionality into your automated email. 

 

Best, 

Courtney Sembler 

Inbound Professor 

Regular Contributor

Have to agree this reply makes no sense. 

 

You enable a/b testing on landing pages, but people visit landing pages at different times of day. 

 

If it makes sense to A/B test ad content or landing pages which deliver at different times of day to individual people, it 100% makes sense to A/B test emails in a flow.

 

It's about improving the flow, and if you are sending a large volume of emails the time of day issue should not skew the data. 

Occasional Contributor

Agreed - this makes no sense and not having A/B test means there is no way at all of testing a change (i.e. a text link or a button link). There may be a point regarding different times of day, but without any form of testing there is nothing at all to test against an we are in the dark of any changes made.

 

Going to try out the following: https://blog.hubspot.com/customers/how-to-a/b-test-workflow-emails-using-the-lead-rotator-or-zapier as some form of A/B testing is paramount to us evolving email 

Occasional Contributor

this is non-sensical. 

this needs to be implemented.

Occasional Contributor

When will Hubspot be implementing this feature? If outreach.io, marketo, etc. can do it, then surely Hubspot can!

-Angela

New Contributor

Hello!

Hubspot representatives, please advise on when A/B testing in automation will be implemented. This is a must-have feature for creating working email automations. 

Occasional Contributor

+1, agreed with everyone else on this.

New Contributor

I have the same issue. I think we constantly need to improve the messages we delivered our customers so A/B testing in automated emails is a must.

Hubspot you should develop that feature

Regular Contributor

I totally agree because we are always updating our workflows to better serve our leads and clients. Even for just the beginning of an email workflow if we had the A/B testing options then we could see which one was performing better and that we could keep for our workflow. We are NOT going to get that info from a one-off email if it is in our workflow. That just doesn't make sense. 

New Contributor

+1. I just started with worfklows, and am quite surprised to see that this feature is missing. It seems like a critical feature to me. Hopefully it will be addressed soon.

Occasional Contributor

Agreed! I had  submitted the same question to Hubspot. Not being able to see which emails are are doing better, what's working whats not is important. These workflows are a big part of my email automation. Checking single emails doesn't give me any insight at all. I specifically create emails that are a part of that workflow only. Also, having ability to see which clients are using a majority of the platform would make my life easier. Right now, there are so many testing browsers. Designing an email in a certain browser may not be optimal for viewing for the other. This is time-consuming and I have to make allowances for a lesser design. So, having that feature which tells me whats the majority of the population using to view these emails - will be a great thing!

 

 

Occasional Contributor

Has there been a solution created yet for this? 

 

It's very manual work if we have to turn on and off emails every time we want to test a new feature, subject line, content etc in a workflow. 
And if I do an A/B test on certain elements in a single-send email, it's not reliable data for me to fall back on since my audiences would have different intents. Help, Hubspot!

Occasional Contributor

I do understand the rationale that automated emails send at different times thus you couldn't definitively say that version A is truly the winner, b/c we can't control time. However, if an automated email has been sent to 300 contacts, we can assume that across the 2 different a/b pools the submission times have averaged out - and thus attribute the success to the corresponding version.

That being said - the proposed option of sending out manual emails until we determine a winner is the least helpful "solution" ever. I'm sure MANY hubspot users have automated workflows that are triggered upon a new contact/lead submitting their information. To have someone on hand, ready to send an email as soon as someone has visited their website and provided their information is ludicrous. FURTHERMORE, they would be sending 1-off emails, and no simple way to actually track and analyze the success of the emails.

 

I still continue to be shocked by the seemingly obvious implementations and functions that Hubspot does not offer.

Currently the no automated email a/b testing option is worse than the option to a/b test automated emails, knowing that the results aren't necessarily 100% accurate.

Hubspot should make this one of their top priorities.

 

New Contributor

I found this on the blog and tried out the first option. https://blog.hubspot.com/customers/how-to-a/b-test-workflow-emails-using-the-lead-rotator-or-zapier

 

Takes a little bit of time to set up, but seems to be working properly so far

Occasional Contributor

We need this badly, any update on when it will be available?

HubSpot Employee

This would be a really great feature! +1 This Idea

Occasional Contributor

Totally +1 this idea. 

Your logic behind not being able to AB test on automated emails is completely flawed - following the same logic, Google should not also allow this kind of tests on their ads!

It's a complete fail on your part, Hubspot, that this functionality is unavailable.

New Contributor

+1 for this Idea... its totally necessary!!

New Member

+1 - Seems so basic.  I saw another tool that handled it beautifully.  Hubspot is a ltech eader and I expected to find it  in the tool.

New Contributor

I hate that HubSpot doesn't offer this feature as well. 

 

I am constantly working to optimize my workflows and the best way I have found to do it is super manual but it has been working. 

 

I write down the day I implemented an email change and monitor it on a monthly basis until it reaches statistical significance using this calculator: https://www.verticalmeasures.com/resources/ppc/statistical-significance-calculator/ 

 

Once the test has reached at least 95% statistical significance I conclude if the test won or was unsuccessful before determining if I will keep this change in place. 

 

It's not the most efficient but it has been working for me! 

New Contributor

I've just set a workflow up using an IF/THEN split and duplicated automated emails with different subject lines (or whatever other element you wish to test).

 

IF "email" starts with "a" through to "m", they get Version 1. If it's "n" through to "z", they get Version 2.

 

It's probably not the best A/B testing methodology, but it should work.

Regular Contributor

Great idea though for now! I might try that and see how it goes. 

Reply
0 Upvotes
GLC
Regular Contributor

I also cannot believe Hubspot do not support this functionality for automated emails or transactional emails. Really need to prioritise this!

Occasional Contributor

I could be late with this but I think this works:

 

https://knowledge.hubspot.com/articles/kcs_article/email/run-an-a/b-test-on-your-marketing-email?KBO...

 

I was able to create an A/B test with 2 different templates and copy. I am yet to test it but so far it looks good.

 

UPDATE: This doesn't solve it, unfortunately Smiley Sad

 

Reply
0 Upvotes
Occasional Contributor

This is really surprising to me. It's really a basic and crucial feature in an automated workflow, I would have expected it to be a part of HubSpot, with the price tag that it comes with.

 

Are there any plans on implementing this?

Occasional Contributor

Yes, it's quite shocking considering we just invested in Hubspot Marketing and I used to use Outreach for Outbound emails.

 

I thought I posted was the solution but it turns out I was wrong as this can only be used with an email list and there is still no way to A/B test in the automated workflow.

 

I will need to figure out how to use Zapier and make a workaround. Insane! 

 

Reply
0 Upvotes
Highlighted
Regular Contributor

+1 here. Other tools have this feature so it makes no sense to not have it on HubSpot. It should be a native function not a zapier/workaround function.

Occasional Contributor

Agreed.

 

They made me some videos on how the workaround works and it involves using two separate workflows rather than just two emails equally tested in one workflow - quite shocking.

 

No timeline was given either for when the feature will be available...

Occasional Contributor

its absolutely ridiculous. this has been an issue for over 2 years and the amount of energy that has to be put in to create the a/b test is assinine. our workflows have as many as 5 emails in them... that would require like 40 if/then/rotators.