Currently the AB test favors Email A. Hubspot has set a 50% win margin for email B to win the AB testing. To be clear, amoung a 1000 contact test pool, Email B needs to generate 750 opens to win the AB test, anything lower that 750 opens then Email A wins. This threshold is much to high, especially when we are testing how to best communicate with contacts through email. Admins should have the ability to adjust the win threshold to what ever value they desire (0%, 5%, 10%, etc). The purpose for an AB test is to determine which email subject line performs best among a common pool of contacts. Currently the test already grants Email A a 75% chance of being utilized. This disadvantage prevents marketers from doing a true AB test.
Definitely agree on that ! It’s very frustrating to see that version A is sent even though version B performed better, but the difference was not big enough for HubSpot. Please allow us to either send to the winning version, even when the difference is slight, or allow us to choose the value we desire to consider a test conclusive.