Creating a direct mail testing matrix

Testing is the path to success in direct mail

Generally, testing involves just two elements – testing one item against another … such as two different mailing lists, or two offers, or two formats or two messages or two of anything else.

This is known as an A/B split test.

You select the two items that you want to test, then mail half with the first item, and the other half with the second item.  The key is to keep everything else in the direct campaign exactly the same.  In that way, you know for sure that any difference in response is the result of the testable item.

You’re not limited to two items of course.  If you want to test 3 lists for example, you split your mailing into three groups and conduct an A/B/C split test.

An A/B/C test would involve testing three of the same thing – that is, three lists, three offers, etc.

This is a fairly common test strategy especially with mailing lists.  It would not be unusual to see a test of five to 10 (or more) mailing lists in one mailing.

But what if you want to test two different elements in your direct mail campaign?

What if, for example, you want to test two lists and two offers?

For that, you would need to set up a direct mail test matrix.

A test matrix allows you test multiple items at the same time.  In the example below, you would split your mailing into four segments,  or test cells – A1, A2, B1 and B2.

As with every test, all other elements of the direct mail campaign are exactly the same.

With this test matrix, you will be able to see several results:

• which combination of list and offer performed the best of four test cells
• which of the two offers produce the better result
• which of the two lists produced the better result

You’re not limited to two offers or lists.  Here’s an example of a test with seven lists and two offers.

But a test of this size can add up pretty quickly in terms of mailing quantities.  A test cell is often 5,000 – 10,000 names (you can test smaller quantities but with less reliability) so a 14-cell test quantity will require a mailing of 70,000 – 140,000 pieces.

What if you can’t afford to mail that many pieces?

Here’s alternative approach.

Notice we removed all but two of the offer tests.  This reduces your total test cells from 14 to 9.

At the same time, you are still able to test the relative effectiveness of seven different lists (using Offer A) and the relative effectiveness of your two offers with smaller quantities.

Why did we select B1 and B6?  It’s a random selection.  Unless one of the lists is the control (an established winner), it could have been any two cells.  If you do have a control in place, you would want include the control list in the offer test because that list has a proven track record with you.

We want to use at least two test cells so that we could verify our results.  Relying on just one test cell could be risky.

If, for example, Offer A beats Offer B in one test cell, but loses in the second test cell, we are left with some doubt.  But if Offer A wins in both cells, we can feel confident in our results.

If it makes you feel any better, use a third test cell.

I have used the list and the offer in these examples in part because they are the most important elements to test.  But you can create a direct mail test matrix for any element of your direct mail or email campaign.