The Art of Testing August 2012

Agenda This is a Test. This is only a Test. Testing in Direct Mail Show Me the Numbers Oooops! Pop… TEST…. Valid or Invalid – you be the Judge!

What Is a test? A procedure intended to establish the quality, performance, or reliability of something, esp. before it is taken into widespread use. A/B testing, split testing or bucket testing compares a control sample to other samples in order to discover how to improve response or conversion rates

What Is NOT a Test?

NO, REALLY - A TEST Is NOT… …..Comparing results of two different campaigns mailed during two different times of the year. A TEST Is NOT… … Comparing results of two mailings at two different organizations. A TEST Is NOT… … Mailing two completely different packages, with different offers, and different asks, with different inserts.

Testing in Direct Mail

Component Test Postage Ask List Offer Who signs the letter? Color Creative Test Package 2 page vs. 4 page letter. Low Risk/ Lower Reward Higher Risk/Higher Reward

Testing in Direct Mail: Knowing When & What to Test When – Quantity is RIGHT Timing is RIGHT There is Rollout potential What – Tests should be designed to meet both station goals and budget. All tests should start with your hypothesis!

Splitting the File RANDOM- Tips on pulling a random file: Team Approach has a Random field! EQUAL: The Test Bucket and the Control bucket need an equal # of records (which may mean a third bucket (fall out) if it is not a 50/50 split! CODED: Each segment needs a unique code for tracking purposes. Dont forget those pesky follow-ups!

Set yourself up for Statistical Success… Final Objective: Statistic validity What makes a test statistically valid? Significance Sample Size What does it mean to be statistically significant? Results are repeatable, not by random chance

Significance Significance measures the likelihood of repeatable results. Determined by mathematical formulas and expressed as a percentage (the confidence level) 95% Confidence Level is best practice Significance tests differ for Response Rates & Average Gifts A winning test does not necessarily mean you have a winning package Free significance calculators are available online

Significance Calculators

Sample Size Why is sample size important? Measuring significance Minimizing risk & controlling costs How do we determine sample size? In general: There is no right quantity to test The larger the sample size, the better Rule of 100 Statistical Calculations

Sample Size: Rule of responses will make this test statistically significant. Some say that this rule-of-thumb has never proved them wrong. Some believe this statement underestimates the actual sample size that is needed.

Sample Size Calculator: Response Rate

Sample Size: Statistical Calculations Response Rate: N = * p * (1-p)/(significant effect)² Average Gift: N = * σ² /(significant effect)² Have no fear! Many free calculators are available online

Rule of 100 vs. Calculations Head to head test… Scenario 1 Control RR: 2% Expected Lift: 25% Calculations show that… Responses needed: 308 Control/Test Quantity: 12,302 Scenario 2 Control RR: 2% Expected Lift: 47% Calculations show that… Responses needed: 102 Control/Test Quantity: 3,481 Results: In scenario 1, the Rule of 100 underestimates the quantity needed. In scenario 2, the Rule of 100 holds true.

Quick Anatomy Takeaways The goal is to set yourself up for statistically valid results from the start Make sure that you have an adequate sample size To determine the right size you need to know the expected response rate, effect, and confidence level. Run the test… One variable at a time; multivariate testing increases sample size. Results are In! Check for significance. Use the calculators to see if your test was significant at the 95% confidence level.

Confidence Intervals Confidence Intervals go hand-in-hand with significance. A range containing the true parameter for the confidence level The true response rate for the population falls within this confidence interval.

OOOPS! Mistakes that Muddle

Mistakes that Muddle 1. Knowing when something just isnt right. 2. Uncovering what that something is.

YOU BE THE JUDGE!

Questions? Thank You!!!