It’s elementary, my dear Watson 🕵🏽

Good day detective, Aelia here.

Now that you know how to run a marketing experiment (ahem, last issue if you missed it!), it’s time to put your detective hat on and check in on your campaigns. 🔍

Trouble is, you might run into the the sweet, sweet trap of vanity metrics.

So what are vanity metrics and how do you steer clear of them so you don’t measure the wrong thing in the first place?

Here is the tricky part: vanity metrics are not the same for every single campaign.

A vanity metric for one experiment might actually be a valuable metric for another!

If you’re going Why, Aelia, must thou hurt me so? in your head, fear not!
For I come bearing the gift of a solution!

Step 1: Define your north star metric(s)

The best way to avoid the trap of vanity metrics is to define a metric that supports the goal of your marketing experiment (it can be more than one if your experiment requires it).

These will be your north star metrics.

How to do this? Simple.

Let’s take an example:

you’re running an email experiment to ask for testimonials from recent customers.

your goal is the action you want your audience to take.

in this case, it’s to get responses with testimonials.

and so your north star metric becomes the number of testimonial responses you get.

everything else—open rates, CTR—becomes background noise, i.e., vanity metrics for this marketing experiment.

Here’s a quick visual for some examples of vanity and north star metrics (and why) for some foundation marketing experiments:

Step 2: Identify warning signs to create a guardrail

Guardrail: identify what can break during your experiment/what can constitute a loss

When setting up your metrics, you might get so absorbed running after your north star metrics that you forget to identify warning signs that your experiment is failing.

And you don’t notice it until you look at measurable impact and go:

For example.
You’re closing deals in a bottom of funnel (read: right before they go ka-ching!) marketing experiment, BUT you’re also leaking leads with high drop-off rates.

Does that make your marketing experiment successful?

Maybe not if those dropoffs are quality leads that left because of a problem at your end. 😬 

or, Example 2,

You get good conversion rate from a paid campaign. but the CPA is too high.
Is it worth it? Probably not.

Especially not if the lead quality isn’t great.

Here’s a visual of some guardrail examples for marketing experiments:

Now, this is not to say you need to panic every time you notice something going wrong.

Just last week, we had a sudden drop in subscribers which left Sophia in a panic of “Waiiiiit, what went wrong?”

But a quick check turned out most of those unsubscribers were personal contacts, not leads. (it’s completely okay if your friends sign up for your newsletters but they end up unsubbing sometime after because it’s not their niche!).

Since you might also run into such issues, I’ve created a handy case file so you can panic only when you need to.

Marketing Experiment Case File

This is what your Marketing Experiment Case File looks like (currently, it’s prefilled with a sample experiment, but you can edit it out when you copy the template):

Now here’s how to use it

Step One: Basic Data

Drop in your basic details:

• Experiment name

• Start date

• End date

• North star metric

• Target (your goal for your north star)

• Guardrails 1,2,3 (drop your target threshold you need to meet here)

Step Two: North Star Check

When your experiment is finished, or when it’s running and you want to do a mid-run check to monitor, check the results of your north star metric. if they are in line with your target, you can move on to the next check for your guardrails, but if the results aren’t great, you might want to stop and investigate here.

Step Three: Guardrails Check

Now, to check your guardrails, see the results for each of them and match them with the threshold you set in the Guardrail columns. Choose “good” or “bad” from the dropdowns to list how your metrics are doing here.

When you’re done with the dropdowns, the “Decision” column will automatically update the status for you, and what you need to do.

Now, take your case file and experiment awayyy!

We’re headed out of office - checks watch right now! - for our our 10-day Agency-wide summer break. We’ll see you and your inbox in two weeks (July 17th)!

Cheers!

Powered by the joy of crocheting a toast-and-egg and blooming sunflowers from Sophia’s kitchen window.

Aelia ⚡️🧕 and Sophia 💜👩🏽‍💻