Email A/B Test 'Winner'

[mwagner21]

I recently ran an email A/B test on two different subject lines. Despite version B having over a 2% higher open rate, that was deemed statisically insignificant and version A was sent by default. From my perspective, a 2% difference in open rate is far from insignificant. So, I would like to either see more clarity into how a winner is determined by the system (which I have not been able to locate, even after speaking with support), as well as the ability to manually select a winner on A/B tests like this. I have used this functionality in other marketing automation platforms previously, and found it extremely beneficial. In that scenario, HubSpot would notify the user who set up the A/B test when the test has completed, and potentially present HubSpot's suggested 'winner'. The user would then be able to either approve HubSpot's suggestion or select the winner themselves. 

[Mike_Donnelly]

Hi @mwagner21 - Totally understand your challenge here and thought you might find the below resources helpful.

One is a statistical calculator (this is a simple one).  While 2% may seem like a big difference (it is), but in order for HubSpot to pick the winner, the results have to work out form a pure math perspective.  In your case, it doesn't seem like it did.

Second, thought this article might help in your efforts.

Lastly, while self-serving, we're a HubSpot app partner and as part of our service, it goes about this in a way that I think provides what you want.  Allows you to take control and pick the winner once you've deemed there is one (statistically significant or gut feel).  If you're up for chatting feel free to reach out to me directly at mike@theseventhsense.com

[mwagner21]

Thanks Jessica! That's what I get for multi-tasking, I thought I was posting in the ideas forum! Ha! 

[JessicaH · Accepted solution]

Hi @mwagner21,

Thanks for reaching out.
Currently, this is not a functionality of the tool. If you would like to see this implemented in the future, I would recommend sharing this idea in our ideas forum.

Take care,
Jess