Every marketer, product manager, or ‘growth hacker’ you’ve ever met has preached to you about the importance of A/B testing in marketing… but it’s ridiculously hard to find any real information about its effectiveness.
Well, I have the information you’re looking for.
I analyzed the results of 330 marketing tests we ran at Ladder for 16 clients – from Y Combinator-backed startups to Global Corporations – to check our failure rate.1
The results will shock you.
Imagine this.
Flip a coin.
Heads? You lose.
Now, imagine you spent hours looking at historical coin flip data, carefully planning your coin flipping strategy, and you (plus the rest of your team) were sure that you’d come up tails.
You were ready to bet your job on the result… and yet… it came up heads.
Damn.
Welcome to modern marketing.
Except the odds are even worse than that coin flip.
According to our data, 57% of marketing tests fail.
We have a grueling interview process involving an Excel test, a copywriting test, a logical reasoning test, a video submission, rounds of interviews, and a (paid) day working on actual client work in our office before we hire someone. (plus we throw away half the applications at random to ensure we don’t hire anyone who’s unlucky…)
We’ve spent a lot of time designing our marketing growth process. Our strategists crunch the numbers before deciding on what marketing growth tactics they want to test. They fill in a test script complete with the hypothesis they’re testing and how they’ll report on performance. They even ‘stress-test’ their ideas in front of a panel of their peers and senior Ladder leadership before they pitch it to the client, who gives their final approval before launch.
Our clients are some of the fastest growing startups and innovative corporations out there. Their users love them, retention is high, and their team executes to a world class level. Despite all the failed marketing experiments, the vast majority of our relationships still drive positive ROI growth.
So what on Earth is going on?
Helping a business grow is a valuable skill. You don’t think Silicon Valley startups pay people with silly titles like ‘Growth Hacker’ over $139,000 a year out of kindness, do you? That’s why Facebook is worth $341b (growth) and Yahoo is selling for $5b (no growth) despite once being worth $125b (when it still had growth).
Even the experts struggle with growth. Noah Kagan, an early employee at Facebook and founder of AppSumo, only sees a 1 in 8 success rate. An industry study by VWO shows a 1 in 7 success rate in business growth. A study by Harvard Business Review pegs the business growth failure rate at 80-90%.
The little known marketing trick that we use? Lose small, win BIG.
How do you lose small? Through A/B testing, which is one of the most important marketing tactics you can use.
Make a small bet by A/B testing something you think will work to grow your business, and only double down on it when you see it working.
Everything is better when you hedge your bets with A/B Testing.
The fact that you even KNOW your failure rate puts you miles ahead of the average marketer, who just does stuff without testing it. They live their lives in blissful ignorance of how many decisions they’re getting wrong, and they never improve.
If their companies grow at all, it’s down to luck.
Smart Marketers make their own luck, by A/B testing.
Don’t blow the whole marketing budget on one ad campaign: start with a few hundred dollars a week and only scale up if it works.
Don’t roll out a new feature to every user: choose a small subsection of users and see if it even moves the needle before rolling it out.
Don’t redesign the homepage ‘just because’: run the numbers to see if it’s a worthwhile exercise vs. all the other actions you could take.
Don’t try every tactic under the sun: model your marketing funnel and pick tactics that can realistically plug the gaps where people drop off.
Don’t make 15 changes at once: systematically test one element at a time to truly learn its relative importance to performance.
Don’t argue about what ad looks the best: take all the different versions and test them in market to let the data decide what’s best.
When you approach business growth with this marketing mindset, every failure is still a big win because you’ve learned another way that doesn’t work.
“I have not failed. I’ve just found 10,000 ways that won’t work.” – Thomas Edison
A person who never made a mistake, never tried anything new. – Albert Einstein
When you adopt this mindset, growth is inevitable. It’s just a matter of time before you find the winning combination.
The truth is that most people never make the effort in marketing to figure out if they’re making the right decisions. Most people just want you to do thing X without ever thinking about whether it’s the right thing to do (“Of course it is! I thought of it and I’m the CEO”) or that maybe thing Y might be a better use of your time.
It ain’t easy: to do this right you have to meticulously keep track of every marketing variable you’re testing and run the numbers to justify each action before you take it. It’s a lot more work and it takes discipline. It’s like losing weight: knowing what you should do isn’t the same as actually doing it.
It’s slow. It’s hard. It’s no fun.
…but it gets results.
Nobody is born with the knowledge of how to grow a business successfully. If your business is a startup, by definition what you’re attempting hasn’t been done before. You just can’t be right 100% of the time… you’d be lucky to get it right 50% of the time.
If you can’t KNOW the right action, HOW can you justify just doing it without testing it first?
Test, Learn, Repeat.
Get smarter about what you test, based on what you learn, every time you repeat the process.
Compounding insights.
If you can do that, and keep doing it, you’ll win.
You get the idea. Whatever your problems in marketing, A/B Testing can help.
*BONUS: Elon Musk even A/B tests his personal relationships. #MaybeTooFar
Sign up to our newsletter to stay up to date with all the latest movements in the field.
More from Ladder Blog
The art world is being disrupted by generative AI, and artists aren’t happy. Generative AI models like DALL-E, Midjourney, and Stable Diffusion were trained by scraping millions of images from the internet, without permission. Artists claim this violates copyright, while the companies training these models argue this falls under fair use.
Read More →In the world of digital marketing and data-driven decision-making, creative testing is a pivotal tool in achieving business growth. Gone are the days of relying on gut feelings or guesswork; now, business decisions are powered by data-validated insights, meticulously collected, analyzed, and validated. This transformative process empowers businesses of all sizes, from established enterprises to budding startups, to thrive in an ever-evolving digital market. This article looks at the practical applications, challenges, and innovative transformations associated with creative testing, offering you valuable insights and actionable strategies to implement in your own digital marketing efforts for achieving growth and success.
Read More →
