July 15, 2021
Every marketer, product manager, or ‘growth hacker’ you’ve ever met has preached to you about the importance of A/B testing in marketing… but it’s ridiculously hard to find any real information about its effectiveness.
Well, I have the information you’re looking for.
I analyzed the results of 330 marketing tests we ran at Ladder for 16 clients – from Y Combinator-backed startups to Global Corporations – to check our failure rate.1
The results will shock you.
Flip a coin.
Heads? You lose.
Now, imagine you spent hours looking at historical coin flip data, carefully planning your coin flipping strategy, and you (plus the rest of your team) were sure that you’d come up tails.
You were ready to bet your job on the result… and yet… it came up heads.
Welcome to modern marketing.
Except the odds are even worse than that coin flip.
According to our data, 57% of marketing tests fail.
We have a grueling interview process involving an Excel test, a copywriting test, a logical reasoning test, a video submission, rounds of interviews, and a (paid) day working on actual client work in our office before we hire someone. (plus we throw away half the applications at random to ensure we don’t hire anyone who’s unlucky…)
We’ve spent a lot of time designing our marketing growth process. Our strategists crunch the numbers before deciding on what marketing growth tactics they want to test. They fill in a test script complete with the hypothesis they’re testing and how they’ll report on performance. They even ‘stress-test’ their ideas in front of a panel of their peers and senior Ladder leadership before they pitch it to the client, who gives their final approval before launch.
Our clients are some of the fastest growing startups and innovative corporations out there. Their users love them, retention is high, and their team executes to a world class level. Despite all the failed marketing experiments, the vast majority of our relationships still drive positive ROI growth.
So what on Earth is going on?
Helping a business grow is a valuable skill. You don’t think Silicon Valley startups pay people with silly titles like ‘Growth Hacker’ over $139,000 a year out of kindness, do you? That’s why Facebook is worth $341b (growth) and Yahoo is selling for $5b (no growth) despite once being worth $125b (when it still had growth).
Even the experts struggle with growth. Noah Kagan, an early employee at Facebook and founder of AppSumo, only sees a 1 in 8 success rate. An industry study by VWO shows a 1 in 7 success rate in business growth. A study by Harvard Business Review pegs the business growth failure rate at 80-90%.
The little known marketing trick that we use? Lose small, win BIG.
How do you lose small? Through A/B testing, which is one of the most important marketing tactics you can use.
Make a small bet by A/B testing something you think will work to grow your business, and only double down on it when you see it working.
Everything is better when you hedge your bets with A/B Testing.
The fact that you even KNOW your failure rate puts you miles ahead of the average marketer, who just does stuff without testing it. They live their lives in blissful ignorance of how many decisions they’re getting wrong, and they never improve.
If their companies grow at all, it’s down to luck.
Smart Marketers make their own luck, by A/B testing.
Don’t blow the whole marketing budget on one ad campaign: start with a few hundred dollars a week and only scale up if it works.
Don’t roll out a new feature to every user: choose a small subsection of users and see if it even moves the needle before rolling it out.
Don’t redesign the homepage ‘just because’: run the numbers to see if it’s a worthwhile exercise vs. all the other actions you could take.
Don’t try every tactic under the sun: model your marketing funnel and pick tactics that can realistically plug the gaps where people drop off.
Don’t make 15 changes at once: systematically test one element at a time to truly learn its relative importance to performance.
Don’t argue about what ad looks the best: take all the different versions and test them in market to let the data decide what’s best.
When you approach business growth with this marketing mindset, every failure is still a big win because you’ve learned another way that doesn’t work.
“I have not failed. I’ve just found 10,000 ways that won’t work.” – Thomas Edison
A person who never made a mistake, never tried anything new. – Albert Einstein
When you adopt this mindset, growth is inevitable. It’s just a matter of time before you find the winning combination.
The truth is that most people never make the effort in marketing to figure out if they’re making the right decisions. Most people just want you to do thing X without ever thinking about whether it’s the right thing to do (“Of course it is! I thought of it and I’m the CEO”) or that maybe thing Y might be a better use of your time.
It ain’t easy: to do this right you have to meticulously keep track of every marketing variable you’re testing and run the numbers to justify each action before you take it. It’s a lot more work and it takes discipline. It’s like losing weight: knowing what you should do isn’t the same as actually doing it.
It’s slow. It’s hard. It’s no fun.
…but it gets results.
Nobody is born with the knowledge of how to grow a business successfully. If your business is a startup, by definition what you’re attempting hasn’t been done before. You just can’t be right 100% of the time… you’d be lucky to get it right 50% of the time.
If you can’t KNOW the right action, HOW can you justify just doing it without testing it first?
Test, Learn, Repeat.
Get smarter about what you test, based on what you learn, every time you repeat the process.
If you can do that, and keep doing it, you’ll win.
You get the idea. Whatever your problems in marketing, A/B Testing can help.
*BONUS: Elon Musk even A/B tests his personal relationships. #MaybeTooFar
Sign up to our newsletter to stay up to date with all the latest movements in the field.
More from Ladder Blog
Get Facebook ad support & answers via our ultimate Facebook FAQ guide – covering ad sizes, approvals, spending, strategy, settings, culture, polit...Read More →
See the marketing plan template that drives our success. This marketing plan example covers strategy, testing, analytics, performance, budgeting, and ...Read More →
Promote a social post to get cheap visits to a website and capture emails. High quality social posts can be targeted to reach specific audiences based on interests, shopping habits, browsing habits, and more. Publish a post on a social network and use their native advertising too...
Place your call to action form or button on the right side of your landing page to increase activation. Focusing on the right side of your landing page rather than the center, especially when the form remains visible while scrolling down the page, keeps registration forms and CTA...