direct response

Creating a direct mail testing matrix

Testing is the path to success in direct mail

Generally, testing involves just two elements – testing one item against another … such as two different mailing lists, or two offers, or two formats or two messages or two of anything else.

This is known as an A/B split test.

You select the two items that you want to test, then mail half with the first item, and the other half with the second item.  The key is to keep everything else in the direct campaign exactly the same.  In that way, you know for sure that any difference in response is the result of the testable item.

You’re not limited to two items of course.  If you want to test 3 lists for example, you split your mailing into three groups and conduct an A/B/C split test.

An A/B/C test would involve testing three of the same thing – that is, three lists, three offers, etc.

This is a fairly common test strategy especially with mailing lists.  It would not be unusual to see a test of five to 10 (or more) mailing lists in one mailing.

But what if you want to test two different elements in your direct mail campaign?

What if, for example, you want to test two lists and two offers?

For that, you would need to set up a direct mail test matrix.

A test matrix allows you test multiple items at the same time.  In the example below, you would split your mailing into four segments,  or test cells – A1, A2, B1 and B2.

As with every test, all other elements of the direct mail campaign are exactly the same.

Offer A Offer B
List 1 A1 B1
List 2 A2 B2

With this test matrix, you will be able to see several results:

  • which combination of list and offer performed the best of four test cells
  • which of the two offers produce the better result
  • which of the two lists produced the better result

You’re not limited to two offers or lists.  Here’s an example of a test with seven lists and two offers.

Offer A Offer B
List 1 A1 B1
List 2 A2 B2
List 3 A3 B3
List 4 A4 B4
List 5 A5 B5
List 6 A6 B6
List 7 A7 B7

But a test of this size can add up pretty quickly in terms of mailing quantities.  A test cell is often 5,000 – 10,000 names (you can test smaller quantities but with less reliability) so a 14-cell test quantity will require a mailing of 70,000 – 140,000 pieces.

What if you can’t afford to mail that many pieces?

Here’s alternative approach.

Offer A Offer B
List 1 A1 B1
List 2 A2
List 3 A3
List 4 A4
List 5 A5
List 6 A6 B6
List 7 A7

Notice we removed all but two of the offer tests.  This reduces your total test cells from 14 to 9.

At the same time, you are still able to test the relative effectiveness of seven different lists (using Offer A) and the relative effectiveness of your two offers with smaller quantities.

Why did we select B1 and B6?  It’s a random selection.  Unless one of the lists is the control (an established winner), it could have been any two cells.  If you do have a control in place, you would want include the control list in the offer test because that list has a proven track record with you.

We want to use at least two test cells so that we could verify our results.  Relying on just one test cell could be risky.

If, for example, Offer A beats Offer B in one test cell, but loses in the second test cell, we are left with some doubt.  But if Offer A wins in both cells, we can feel confident in our results.

If it makes you feel any better, use a third test cell.

I have used the list and the offer in these examples in part because they are the most important elements to test.  But you can create a direct mail test matrix for any element of your direct mail or email campaign.

Previous Article
Lean Startup

Lean Startup: 4 Primary Principles to Boost Production

Next Article

Why 75% of All Startups and New Products Fail

Related Posts
Read More

The Role of a Recruiter and HR in Small Business

You’ve launched your business and it’s humming along. Like most entrepreneurs, you wear plenty of hats, including chief human resources (HR) and recruitment officer. Here’s the problem, though: You can’t handle all your employee-related responsibilities forever. If you do, you could find yourself in trouble. The issue isn’t just that you’re going to spread yourself...
hiring mistakes
Read More

Hiring Mistakes You Could Be Making

Bad hires happen. However, they happen a bit too often. Studies show that nearly three out of four employers admit to hiring the wrong job candidate. Maybe HR was in a rush to fill an important role or you hired someone because you knew the applicant and they seem nice enough. Or, someone didn’t take...
quality employees
Read More

More Money Isn’t the Only Thing That Will Attract Quality Employees

As a business owner, you are constantly battling with how to boost profits and productivity without increasing costs. Commonly, the first adjustment that comes to mind is within your own team. Finding quality staff members is valuable, though these workers often come at a higher price.  Paying your staff more than your competition can certainly...
relational intelligence
Read More

5 Ways to Retain Talent by Using Relational Intelligence

The following is adapted from "Relational Intelligence: The Five Essential Skills You Need to Build Life-Changing Relationships" by Dr. Adam C. Bandelli. Stress affects all of us differently. Since the start of the COVID-19 global pandemic, employees have had to deal with various challenges in both their personal and professional lives. The lines between our...