One-time A/B Emails in Workflows

[MAlt2020]

I want to send a series of A/B emails over the course of one week in a workflow.

According to the Hubspot Article on A/B tests in workflows, 

"Over time, HubSpot will send the email variations across enrolled records. The 50/50 split may not occur immediately. This split is gradual and will occur over time as the number of recipients increase".

Since I'm only sending each email once, I'm now worried that the 50/50 split wont apply to the email. Can someone tell me if it still does? Thank you!

[karstenkoehler · Accepted solution]

Hi @MAlt2020,

Generally yes, but it depends on your email volume. If your workflow will deliver only 10 emails in that week,  there's a chance it won't be 5/5 but maybe 6/4 or 7/3. It's hard to say for sure. In any case, for that small of a sample size you're not going to get a statistically significant result anyway unless one of the variants is drastically better than the other. You can play around with volume and conversions here to get a feeling for statistical significance: https://www.surveymonkey.com/mp/ab-testing-significance-calculator/

What the article is referring to is that the split is random and randomness will become more 'even' the larger the numbers are.

If you're sending hundreds or thousands of emails in that week, you shouldn't be concerned about the split being off by much. You might not be looking at 50/50 but maybe 51/49 – I haven't tested it in that much detail but that's what I'd expect.

In other words: This is only an issue if your sample is so small that you're likely not going to see conclusive results anyway. If the sample is big enough, then both issues will be solved – statistical significance and an even distribution.

Best regards!

[MAlt2020]

Thank you so much for your swift reply! We'll be sending several thousands, so thats a bíg enough sample!