The Reality of A/B Testing | Case Studies of A/B Testing at Ladder
A/B Testing
Case Study

The Reality of A/B Testing | Case Studies of A/B Testing at Ladder

Michael TaylorMichael Taylor

July 15, 2021

A/B testing in real life.

Every marketer, product manager, or ‘growth hacker’ you’ve ever met has preached to you about the importance of A/B testing in marketing… but it’s ridiculously hard to find any real information about its effectiveness.

Well, I have the information you’re looking for.

I analyzed the results of 330 marketing tests we ran at Ladder for 16 clients – from Y Combinator-backed startups to Global Corporations – to check our failure rate.1

The results will shock you.

Imagine this.

Flip a coin.

Heads? You lose.

Now, imagine you spent hours looking at historical coin flip data, carefully planning your coin flipping strategy, and you (plus the rest of your team) were sure that you’d come up tails.

You were ready to bet your job on the result… and yet… it came up heads.


Welcome to modern marketing.

Except the odds are even worse than that coin flip.

According to our data, 57% of marketing tests fail.

Why do more than 50% of marketing tests fail at moving the needle?

It’s not the people in marketing…

We have a grueling interview process involving an Excel test, a copywriting test, a logical reasoning test, a video submission, rounds of interviews, and a (paid) day working on actual client work in our office before we hire someone. (plus we throw away half the applications at random to ensure we don’t hire anyone who’s unlucky…)

It’s not the marketing process…

We’ve spent a lot of time designing our marketing growth process. Our strategists crunch the numbers before deciding on what marketing growth tactics they want to test. They fill in a test script complete with the hypothesis they’re testing and how they’ll report on performance. They even ‘stress-test’ their ideas in front of a panel of their peers and senior Ladder leadership before they pitch it to the client, who gives their final approval before launch.

It’s not the marketing products…

Our clients are some of the fastest growing startups and innovative corporations out there. Their users love them, retention is high, and their team executes to a world class level. Despite all the failed marketing experiments, the vast majority of our relationships still drive positive ROI growth.

So what on Earth is going on?

The truth is… growing a business is hard.

Helping a business grow is a valuable skill. You don’t think Silicon Valley startups pay people with silly titles like ‘Growth Hacker’ over $139,000 a year out of kindness, do you? That’s why Facebook is worth $341b (growth) and Yahoo is selling for $5b (no growth) despite once being worth $125b (when it still had growth).

Even the experts struggle with growth. Noah Kagan, an early employee at Facebook and founder of AppSumo, only sees a 1 in 8 success rate. An industry study by VWO shows a 1 in 7 success rate in business growth. A study by Harvard Business Review pegs the business growth failure rate at 80-90%.

How to grow a successful business through A/B testing

The little known marketing trick that we use? Lose small, win BIG.

How do you lose small? Through A/B testing, which is one of the most important marketing tactics you can use.

Make a small bet by A/B testing something you think will work to grow your business, and only double down on it when you see it working.

Everything is better when you hedge your bets with A/B Testing.

  • You aren’t ‘betting the house’ on every decision. You’re making lots of small bets that won’t hurt you if they fail.
  • You’re tracking the results of your actions so you know which tactics to double down on (and which to abandon).
  • You’re actually learning what works (or not) rather than hoping and guessing, and you just get better, faster.

The fact that you even KNOW your failure rate puts you miles ahead of the average marketer, who just does stuff without testing it. They live their lives in blissful ignorance of how many decisions they’re getting wrong, and they never improve.

If their companies grow at all, it’s down to luck.

Smart Marketers make their own luck, by A/B testing.

A/B testing examples.

Don’t blow the whole marketing budget on one ad campaign: start with a few hundred dollars a week and only scale up if it works.

Don’t roll out a new feature to every user: choose a small subsection of users and see if it even moves the needle before rolling it out.

Don’t redesign the homepage ‘just because’: run the numbers to see if it’s a worthwhile exercise vs. all the other actions you could take.

Don’t try every tactic under the sun: model your marketing funnel and pick tactics that can realistically plug the gaps where people drop off.

Don’t make 15 changes at once: systematically test one element at a time to truly learn its relative importance to performance.

Don’t argue about what ad looks the best: take all the different versions and test them in market to let the data decide what’s best.

The Growth Mindset All Marketers Need To Follow

When you approach business growth with this marketing mindset, every failure is still a big win because you’ve learned another way that doesn’t work.

“I have not failed. I’ve just found 10,000 ways that won’t work.” – Thomas Edison

A person who never made a mistake, never tried anything new. – Albert Einstein

When you adopt this mindset, growth is inevitable. It’s just a matter of time before you find the winning combination.

The Sad Truth About Marketing: We’re Lazy.

The truth is that most people never make the effort in marketing to figure out if they’re making the right decisions. Most people just want you to do thing X without ever thinking about whether it’s the right thing to do (“Of course it is! I thought of it and I’m the CEO”) or that maybe thing Y might be a better use of your time.

It ain’t easy: to do this right you have to meticulously keep track of every marketing variable you’re testing and run the numbers to justify each action before you take it. It’s a lot more work and it takes discipline. It’s like losing weight: knowing what you should do isn’t the same as actually doing it.

It’s slow. It’s hard. It’s no fun.

…but it gets results.

Growing A Business Is Harder Than It Seems

Nobody is born with the knowledge of how to grow a business successfully. If your business is a startup, by definition what you’re attempting hasn’t been done before. You just can’t be right 100% of the time… you’d be lucky to get it right 50% of the time.

If you can’t KNOW the right action, HOW can you justify just doing it without testing it first?

Test, Learn, Repeat.

Get smarter about what you test, based on what you learn, every time you repeat the process.

Compounding insights.

If you can do that, and keep doing it, you’ll win.

Solve your marketing problems by A/B Testing

  • Nobody using your product? Good. Start A/B Testing.
  • Competitors stealing customers? Good. Start A/B Testing.
  • Is your growth stagnant? Good. Start A/B Testing.
  • Are your targets too high? Good. Start A/B Testing.
  • Want that promotion? Good. Start A/B Testing.
  • Chasing a funding round? Good. Start A/B Testing.

You get the idea. Whatever your problems in marketing, A/B Testing can help.

*BONUS: Elon Musk even A/B tests his personal relationships. #MaybeTooFar

  1. I used a sample of 330 marketing tests from 6 months in 2016, across 16 clients. 144 had a negative impact, 42 were inconclusive (treated as a failure) and 144 had a positive impact (statistically significant to a 95% confidence level).

Ladder powers strategy and performance solutions for fast-growing brands

Talk to a strategist →

Growth is a high velocity game.

Sign up to our newsletter to stay up to date with all the latest movements in the field.

More from

796+ open sourced growth tactics used by the best marketers in the world.
Explore our playbooks →