I recently ran an email A/B test on two different subject lines. Despite version B having over a 2% higher open rate, that was deemed statisically insignificant and version A was sent by default. From my perspective, a 2% difference in open rate is far from insignificant. So, I would like to either see more clarity into how a winner is determined by the system (which I have not been able to locate, even after speaking with support), as well as the ability to manually select a winner on A/B tests like this. I have used this functionality in other marketing automation platforms previously, and found it extremely beneficial. In that scenario, HubSpot would notify the user who set up the A/B test when the test has completed, and potentially present HubSpot’s suggested ‘winner’. The user would then be able to either approve HubSpot’s suggestion or select the winner themselves.
+1
So agre with you, 4 years later I am going through the same problem ! and I agree with you idea ..
I so agree - just had a chat with HS and I am taken aback that this is still not possible. I used to work for an ESP where already in 2008 we had this, and many more user-friendly options - well, we even had ABC or ABCD tests… multivariant.
Agree. It’s disappointing they don’t have this functionality still when other systems do. It should pick the one that has the higher rate as the fallback regardless, and just tell you that the test results were not significant. Or give an option to manually select the winning version at the end of the test timeframe. I did find something from Hubspot that says for most tests it has to be statistically significant at 95% confidence. Based on my calculation for the test I’m running right now, I’m at 90% confidence, and unfortunately, the lower performing one is the fallback version. It’s literally a couple opens away from hitting 95% confidence.