Machine Learning for Marketers: We Built a Marketing Tactic Recommendation Engine

*’Machine Learning for Marketers’ was updated March 2019 with the recent release of our automated marketing audit technology.

Here’s how we see it at Ladder — Historically, marketing has been predominantly driven by guesswork and luck based on what marketers think or have heard/read works best. But guesswork doesn’t lead to a repeatable, scalable model for growing a business.

Over the years since we started Ladder, we’ve been working to remove that guesswork from growth, bit by bit.

In 2014(ish), we validated a repeatable growth marketing process.

In 2016, we built a proprietary test management platform to empower our growth strategists to build and execute marketing experiments – a drag-n-drop sprint builder that allowed for full transparency into what marketing tests are live, paused, off, or yet to launch… as well as the hypothesis, performance notes, and assets in play (creative, messaging, audience).

Pretty cool right?

*NOTE: The beauty behind our test management platform was that it saves data on tests launched, how often they conclude successfully, and what company attributes are associated with the account in which it’s launched. This was the foundation needed for our eventual recommendation engine.

In 2016 we also built and released the world’s largest database of growth marketing tactics.

The free database saw some traction on Product Hunt (see The Ladder Growth Playbook) and has since driven thousands of leads.

In 2017, we knew the major next step for us in removing guesswork was to build proper machine learning directly into our proprietary technology to enable our team to make better strategic decisions driven by data.

We have a deep database of information about what marketing tactics have been successful, the ROI they’ve driven for clients, the industries and business models that they work on, and much, much more.

As a forward-thinking growth agency, we have always valued product development and the productization of our methodology/insights. This being the case, we happen to have a very talented product team with a functional knowledge of machine learning, so we built a basic version of machine learning into our platform. This machine learning v1 powered a baseline recommendation engine for marketing tactics.

With our MVP in place, and seeing validation (and excitement) internally, we moved to take our marketing recommendation tactic engine to the next level with full machine learning implementation (with help from team at Data Revenue).

⇒ Access Free Tactic Database ⇐

Hiring A Machine Learning Consultancy

Edwin Plotts, now Ladder’s director of growth, met Data Revenue (a machine learning consultancy) at a startup event in New York City. Seeing real potential in a partnership, he connected them to Ladder’s co-Founder/COO, Michael Taylor, and CTO, Adrian Slipko.

Compared to other vendors/consultancy we had spoken to, Data Revenue stood out a genuine and collaborative.

The biggest positive sign was that they were straightforward in understanding our goal: Building a machine learning foundation today that will become superior tomorrow in the long run, as we continue to gather more data from our work with clients.

Other vendors tried to sell us the dream and didn’t really come back with any hard information about what machine learning models they would use, while Data Revenue listened to our goals and provided model recommendations based on those goals.

So big shout out to Data Revenue 🙂

A Marketing Recommendation Engine

For Data Revenue, the task was straightforward:

Build a machine learning system using algorithms to recommend the right marketing tactics to strategists and companies, knowing the right tactic for the right company at the right time. Beyond that, based on feedback, recommend even more tactics.

In interviewing Markus Schmitt, the founder of Data Revenue, it turns out that what we had already was a strong baseline to work off of — we were already properly tracking a lot of pre-recommended tactics for a significant number of clients. Props to our own product team 🙂

The question for his team was “do we have enough data to take a machine learning algorithm and let it learn which tactics work with other tactics?” We wanted a Netflix-style recommendation engine…what you viewed, what you liked viewing, what else might you like, etc.

How we do strategy for any company isn’t by choosing a tactic at random. We look at the data and recommend strategy both based on an understanding of the company and on what tactics drove ROI in the past.

So, our machine learning implementation ensures that every time intelligence is input into system, the recommendation engine can use that to better automate relevant tactic suggests.

Implementing Machine Learning Models

It took Data Revenue 4 weeks to complete the implementation of new machine learning to create a full marketing recommendation engine for the Ladder’s database. After testing a few different algorithms, they found the best and most elegant approach for Ladder (that could also be expanded on in the future) would be an open-source model called LightFM.

According to Markus, evaluating a recommender system is not trivial. Data Revenue built custom evaluation functions that can tell you how good a recommender is without having to plug in a full dataset in order to test LightFM.

In our case, the test ran to see whether the algorithm was learning to rank the right recommendation types based on data inputs.

The final check was a subjective evaluation process that involved our co-Founder, Michael, performing a double-blind test on whether these recommendations would be ones he’d consider high-quality himself.

Data Revenue gave Michael the system’s recommendations. He looked at the existing tactics used and the ones the algorithm recommended without knowing the name of the company, and asked himself “Would I have recommended these tactics based on the prior successful tactics?”

The accuracy of the algorithm was impressive: 95% of the time, Michael would have chosen the recommended tactic.

A second check involved revealing the company name and looking at the recommendations again. Once again, 95% of tactics were ones he would have chosen.

Note: For Michael, the evaluation was whether each individual tactic was a good fit for the company. If he were to just come up with 10 tactics, they wouldn’t have been 95% the same. The algorithm is meant to answer the question “Are we recommending tactics that could work for this client?” From there, we can get more intricate with recommendations as the algorithm evolves.

Finally, everything was ready for us to fully implement the machine learning system into Ladder’s growth technology.

^Above is a look at one of the tactic cards from our database, The Ladder Playbook. You can see all the data points that are feeding the recommendation engine: tactic descriptor tags, company attributes, success rates, etc.

⇒ Access Free Tactic Database ⇐

Upgrading the Recommendation Engine

Our CTO, Adrian, worked directly with Data Revenue throughout the entire process, from creation to testing to implementation – in order to ensure we could easily implement their work into our growth platform.

His first step was to provide them with 100% anonymized data they could use to build the system. Data gathering was exported straight from our database, and he didn’t have to spend a long time preparing data for them. They just used what we had in our database, with no client and user names associated.

From that, they used the data we provided to pick out the best algorithm, and then had a discussion about how to implement it into our technology. The result was an API that, when a company ID is given and a number of recommendations to display was set, the system returns recommended tactic IDs.

Below is a look at the resulting Netflix-style marketing tactic recommendation page in our proprietary platform:

*NOTE: As of today this platform is proprietary – as in not publically accessible – however it is a hell of a perk in choosing Ladder as your agency partner 😉

The final product was built as a microservice that is hosted on Heroku and has an API that can be called, asking for next-best recommendation. The API is usable for anything on our side, from enriching recommendations in-app, to building minimal apps that surface top 20 recommendations for a strategist, and much more.

The Future of Automated Marketing Intelligence

At the time of implementation, our database (The Ladder Playbook) needs to be manually updated with new tactics as we discover/create them. Consider we have over 800+ tactics tagged and ready to explore, it hasn’t bee to much of an issue. However, the world of marketing and growth evolves every day and with it new tactics are popping up all the time.

Ultimately, we’re working to automate updating of the database.

How? By creating an auditing system that automatically analyzes and summarizes growth insights across any channel. Part of the auditing process it to automatically pull all assets and activity histories for tagging…which will fuel the database and recommendation engine.

Automated Audits, Insights, & Marketing Intelligence

Good news – we’ve actually already built an intelligent auditing system.

Spotlight is the next core piece in Ladder’s mission to remove the guesswork from growth. It combines 10+ industry leading tools into one intelligent system so that you can fully automate all marketing audits… from Google Ads and SEO, to competitor analysis and analytics.

As of today, Spotlight also includes a machine learning algorithm that automatically tags images pulled from the audits with descriptors. A building block for ultimately feeding our technology tag-based performance and auto-populating the tactic database.

It also allows you to access real data-science at your fingertips in the form of drag-n-drop widgets for anomaly detection and plotting efficient frontier (e.g. “for every $1k increase in spend, you can expect a $4.35 increase in CPA”):

Our product roadmap has us aiming to have a continuously updated tactic database and recommendation engine, fueled by automated audits across every marketing channel, by Q1 2020.

High aspirations, but considering our general strategy success rate is 300% above average I’d say we have a better chance than any other company. And that’s exciting as hell.

If you’re looking to implement machine learning into your own product, we’d highly recommend you use a service like Data Revenue to dip your toes in the water and get started today. You’ll learn a lot, and without taking on too many fixed costs – setting yourself up for future success.

Interested in growing your business via marketing strategy that’s powered by data, technology, and machine learning?