1. wood for trees 1wood for treeswww.woodfortrees.net 1 Outstanding Marketing Services Provider 2011 (Database Marketing awards); Supplier of the Year 2012 (IoF SIG Awards); Innovation on Data Quality 2012 (Database Marketing Awards) Experimental Design in the Commercial Sector Wood for Trees March 2015

2. wood for trees 2wood for treeswww.woodfortrees.net 2 Background – Capital One Founded by Richard Fairbank and Nigel Morris 1988 Massive Growth - now 8 th Largest Bank in US Revenue US$ 24 Billion Net Income US$ 4 Billion 40,000 Employees USP = Information base strategy 100,000s test per year "Using scientific testing on a massive scale, we gather huge amounts of information to help us tailor products and services to the individual consumer, rather than simply offering one product to broad socioeconomic groups. We don't believe that 'one size fits all'.

3. wood for trees 3wood for treeswww.woodfortrees.net 3 Background – Traditional Test Methods UK employed traditional A / B Testing Success in some cases, but issues too: Large number of tests Relatively slow process Documentation Lack of consistency No consideration for impact of tests on different groups of supporters Feedback from Stakeholders around the business was not great …

4. wood for trees 4wood for treeswww.woodfortrees.net 4 Background – Stakeholder Feedback How should I combine multiple tests? We Keep Retesting the same things …. I can’t remember or find out what we’ve test before Is this really working? There seems to be different ways of analysing the tests The tests are too large for us to do many of them The test do not take into account the different types of supporter

5. wood for trees 5wood for treeswww.woodfortrees.net 5 Solution – Introduce a New Framework Statistics Team would take responsibility for testing as a whole Roll out of a New “Testing Framework” Based on DOE Agreement from Senior Management Workshop for all Marketing Teams Specialist support All Tests required formal approval Analysis Team Marketing Stakeholders

6. wood for trees 6wood for treeswww.woodfortrees.net 6 New Framework – Stakeholders Responsibility Required to complete a Form for each test Hypothesis – objective Desired Outcome Power of test Contact information – for results Approval Documented and stored by Analysis Team

7. wood for trees 7wood for treeswww.woodfortrees.net 7 New Framework – Analyst Responsibility Move away from existing “OFAT” approach Considered inefficient (sample size) and ineffective (interactions not tested) Tools for planning (sample size and power) Design Analysis Required more analysis (and resource) post campaign But in theory analysis could be shared between business lines and statisticians

8. wood for trees 8wood for treeswww.woodfortrees.net 8 New Creative Normal Creative Freebie10,000 No Freebie10,000 DOE – Benefits (Sample Size) Freebie (20,000) No Freebie (20,000) VS New Creative (20,000) Normal (20,000) VS Assume that we want to test 2 factors: Freebie and New Creative Standard sample size calculation suggest that 20,000 is needed in each cell for OFAT Test For DOE we would only need half the size! OFAT DOE

9. wood for trees 9wood for treeswww.woodfortrees.net 9 DOE – Benefits (Interaction) Model allows for Interaction between factors to be examined In this hypothetical example the biggest increase in response rate is found by combining Freebie and the New Creative Creative A vs B Response Rate Freebie No Freebie Testing OFAT basically ignores the fact that different elements may work together A B

10. wood for trees 10wood for treeswww.woodfortrees.net 10 DOE – Benefits (Covariate Analysis) The other key question is does the effect of the envelope size and free gift vary for different groups of supporters? This can be analysed in a similar way as to the interaction between variables VS

11. wood for trees 11wood for treeswww.woodfortrees.net 11 DOE – Analysis Finally, this could be analysed formally using a form of the General Linear Model: It is possible to analyse some simpler designs “by hand” However the approach taken at Capital One was to adopt the more statistical approach using appropriate software

12. wood for trees 12wood for treeswww.woodfortrees.net 12 DOE – Considerations It requires a certain level of statistical knowledge and the right software Even then it is a very deep subject: how many factors can be tested, what level of interactions are needed, fractional factorial design and d-optimal design … … Plus it can be hard for stakeholders to appreciate and is sensitive to execution issues

13. wood for trees 13wood for treeswww.woodfortrees.net 13 Summary What went well? Senior Management buy in Standardised process – documentation, consistency Workshops What can cause problems? DOE complex and not understood by all Sensitive to operational issues Other Take-outs DOE is the most powerful way to ‘learn’ ?! Internationally recognised (six sigma) More success stories outside of manufacturing and agriculture – such as web site design (quitting smoking)