A/B Testing send times on a daily email newsletter

Marketing & Content Hub Email Marketing Tool
1

We send a daily editorial email newsletter out of Hubspot. We’re considering doing an A/B test where we would test send time. We would test sending at 9am for 50% of the list (this is the time the newsletter goes out currently) and then the other 50% of the list would get the newsletter at 1pm. We would run this test for a full week, Monday-Friday.

My concern is since this test would be randomized on a 50/50% basis every day, it’s possible someone will get the newsletter at 1pm on Monday, for example, and then Tuesday would get the newsletter at 9am. I’d be concerned that this would skew results--they may be less likely to open the newsletter on Tuesday since they feel that they “just got” the newsletter on Monday. Would it make more sense to leave the lists the same for each day of the test, so that half of the list gets the newsletter at 1pm everyday of the test while the other half gets the newsletter at 9am everyday of the test?

Are there best practices to follow in executing a test like this? Thanks.

2

Hey Brian,

I’ve found the best way to do this is a little bit manual.

1.) Create a Contact Property called “Test Groups” with a dropdown select and A, B, C, etc for the number of groups you want to test.

2.) Export all contacts in the newletter list and create a column in the CSV with Test Groups and add A,B,C, etc evenly among the contacts.

3.) Re-import the contacts, allowing the Test Groups field to be updated.

4.) Clone and send emails to the groups separately. That will control who gets it when so they don’t randomly switch.

5.) If new subscribers come in, have a separate List for “Email Subscribers with no Test Group” and send to them outside with a different email.

I did a two month long test with a client that landed the best send time for their batch email audience at 2 p.m. on Wednesdays. No one saw that coming.

3

This is so manual - this should 100% be a feature in HS. We need to know what times work best for our leads and clients so we can really personalize their experience with us.