AMA: Running Marketing Experiments [Now Closed]

[lalexander]

This Thursday, a HubSpot resident expert on running marketing experiments will be answering your questions on conversion rate optimization, experiment design, A/B testing, and data analysis.

Alex Birkett is a Senior Growth Marketing Manager at HubSpot who focuses on user acquisition. He runs his own content marketing agency and specializes in growth strategy, data analysis, experimentation design, and SEO. He previously worked at ConversionXL - an industry leader in optimization and growth. In short, he’s got a lot of experience to share with you!

Update: This AMA is now closed.

Some things you can ask Alex about:

1. How he’d approach optimizing a conversion path on your website
2. Tips for setting up an experiment you’re considering
3. Advice on which experiment to run next to maximize your return
4. How to skill up in experimentation and get your team in an experimental mindset, too

If you read someone else’s question or comment and have thoughts to share, please jump into the conversation and offer your two cents, too! We welcome collaboration.

[lalexander]

I remember there were a couple of folks during a workflows feedback session last month who’d said they were interested in this AMA. Tapping you in, @Rafael_Marín and @EnricoGiardini!

[shilpandya]

@RiccardoPisani @ridingforlife @baylistefl @LBeresford-Ward @rachma @jtAccusoft @cynthia_dunlop @willsmith

Hi all, wanted to make sure you all saw this opportunity to ask Alex your testing and optimization questions. Are there any challenges your company or clients are facing with conversion rates or website performance? Drop your questions in this thread!

[iamalexbirkett]

Hi everyone - excited to answer questions today! Anything on conversion optimization or experimentation is welcome - this could be the behavioral psychology side of things, the cultural components of building experimentation / growth teams, or the analytics side of designing and analyzing tests. Happy to help on any and all CRO/experimentation topis!

[Alex27]

@iamalexbirkett Thank you for your time today.

Not sure if you have ecommerce experience… but worth a shot. What are the most effective experiments you’ve run on ecommerce sites? There is a lot of talk about changing button colors, checkout flow, navigation, etc. and sometimes our team is paralyzed because we don’t know where to start.

Where would you start? What considerations go into that prioritization?

[iamalexbirkett]

Hey there @Alex27 - thanks for the question! Lots of ecommerce experience from my CXL days.
First, the gold standard is conversion research. Instead of thinking about elements or tactical components, you may want to step back and diagnosis the biggest user experience bottlenecks as well as the highest impact opportunities. Typically, you do this through a mixture of quantitative (Google Analytics, heat maps, etc) and qualitative research (surveys, interviews, user testing). You can then detect patterns and map out high impact potential based on how many users will be exposed to the new experience and how well/poor it is currently performing. You’ll typically then slot these findings into some sort of prioritization matrix (there are many - PIE model, ICE model, but my favorite is the PXL model,).
Second, in a slightly more domain-specific answer, ecommerce experiments tend to be more valuable the closer they occur to the transaction. This means analyze your checkout funnel and find the biggest drop-off points, run user tests, and try to iron out any friction in the checkout experience. Then you can back out to the cart level, then the product page, then the category, etc.
Third, there are ‘patterns’ as to what works well, but there’s variance as well. Will depend heavily on your industry and customer base. That said, defaulting to behavioral psychology tenets is a good way to start ideating. With past CPG clients, social proof worked particularly well (I remember I implemented Fomo - https://fomo.com/ - for an ecommerce client that sold high end candles back when the Fomo notifications were a novel thing and it crushed it). At a general level, a good framework for analyzing pages for persuasion/conversion potential is the LIFT model from WiderFunnel.

Finally, the returns on marginal changes tend to be marginal. At scale, those numbers and the cost/benefit tradeoff make sense (it takes 10 seconds to change a button color and a 1% increase can mean a lot with millions of visitors). These case studies are popular because they’re easy to illustrate in the context of a/b testing and sites with lots of traffic can detect small lifts. Most companies have to ‘swing bigger,’ though. Typically prominent changes and bigger tests based on user behavior insights do the trick.

[AM8]

Hi Alex,
My question is:
Companies want ‘growth marketing’ with content and SEO but also want instant leads with paid platforms. The team working on creating blogs and content and promoting content organically sees growth at a lesser speed compared to the growth in leads from the paid channels in the short term. There is simply no ‘instant gratification’ in organic content strategy and SEO. It is true that paid growth stops as quickly as it started the moment we stop pouring money into paid channels and while the senior leadership does push for sustainable organic growth, managers are often quick to ‘replace’ folks working on organic growth content strategy by saying that they are not bringing in results. What are your thoughts on SEO and content taking ‘time’ to show results and how to create some initial wins?

[iamalexbirkett]

@AM8 This is a phenomenal question.

I’m approaching this from a few different angles. First, my core inclinations bias towards the rapid experimentation / rapid feedback loop channels, so I love the ability to run quick smoke tests as well as the ability to splice and evolve your creative at rapid speed with paid channels. However, I both work at HubSpot (which clearly believes in content and organic) and I run a small content agency. The latter experience in particular leads me to a high degree of empathy with your situation, as I’m constantly in conversations that sound just like what you described.

“Can we just try content for 1-2 months and then determine if we want to continue investing?”

Hate to say it, but statements like this are a red flag. It’s difficult to change someone’s idea of what an attribution window should feasibly look like. For paid ads, it could be days or less. For content, particularly of the SEO-drive variety, it takes months to see results, which start with leading indicators (we’re starting to rank), crest into vanity metrics (we’re getting some traffic!), and eventually turn into truly attributable business results (we can tie back $X to content as a channel, and we can determine the estimated value of each article we’ve written).

The problem with organic/content is the flip side to its biggest strength; it’s a true flywheel, but you’re building up 80% of the iceberg before it ever peaks through to the surface. So you have to look at the two channels fundamentally differently, typically laying the groundwork for organic as early as possible so the flowers can sprout months down the line. Eventually the two have a synergistic effect as content visitors give you greater remarketing granularity and ads data gives you great insights on what keywords and audiences convert well, but from an investment standpoint, one is buy and hold and the other thrives on volatility and fast adaptation.

As for quick wins, index heavily on content promotion. Do what you can do write “buzzworthy content,” especially in the beginning, stuff that garners links, social shares, and attention. Traffic spikes aren’t going to pay the bills, but they’ll buy you time to invest in the longer term plays. Content promo obviously isn’t easy or a silver bullet either, but there are certain channels and tools that are high probability (Quuu and paid promotion tools, influencer outreach done well, and engaging in communities in your niche are all tried and true). The benefit is that this ‘buzzworthy content,’ provided it builds you some backlinks, will also end up helping your more organically-inclined content plays as well.

Anyway, that’s a long way of saying the following: quick wins are feasible but still mildly unpredictable and investment in organic/content is mainly an education/buy-in issue. As for the agency, we typically drop the conversation pretty early if the client is too pushy on seeing results within weeks or what we deem as an unreasonable time frame.

A really great article that Nat Eliason wrote on the topic here: https://www.growthmachine.com/blog/critical-authority-threshold
I also tried to touch on this a bit in my big post on content marketing strategy (which was written with startups in mind): Content Marketing Strategy: Everything You Need to Know to Build a Growth Machine - Alex Birkett

[AM8]

@iamalexbirkett
Wow! I never expected such a detailed response. Thank you for taking my question seriously and for taking the time to respond. Your tips and insights are really helpful and actionable.
It is true that organic+content is a buy-in/education thing, but with my experience of watching the paid team succeed, I was beginning to doubt organic and content. However, thanks to your post I will continue to focus on what I know is the more sustainable healthy growth for business while at the same time improving on paid on an ‘as-needed’ basis.
Thank you again!

PS: you have a new subscriber on your blog! (Great content there too)