1. (Really) Knowing Your Customer with Big Data Hugh E. Williams hugh@hughwilliams.com @hughewilliams http://hughewilliams.com

2. Test versus control experimentation Divide the customers into populations One (or two) population is the control One or more populations are the tests Collect vast amounts of data from each population Compute metrics from the data, including confidence intervals Understand the results Make decisions

3. Example: Larger Images in eBay’s Search

4. Test versus control experimentation… It’s hard to segment the population: – Many users aren’t logged in – Many users use several browsers – Many users use several devices Attributing a result to a change is hard – Need to know that the user was affected by the change – Need to understand interaction effects of changes – Metrics are noisy Decisions are made with a confidence that isn’t 100%

5. What won’t data tell you? (Usually) An unambiguous picture What you don’t measure Exactly what to do How to leap from one mountain to another When to take risks When to be decisive

6. THANK YOU PS. I’m available for advising and consulting. hugh@hughwilliams.com