Inside the Mind of the 21st Century Customer Alan Page

Evaluating a Scenario Ease of use Responsiveness UsefulnessVisual Appeal

Example: Showcase (marketing speak) Scenario (project manager speak) End-to-End Test (something actionable) (Optional) Variations Xbox Knows You Better Identity Step in front of the console, get recognized and see my curated content Light/Dark settings Child/Adult/Gender/Height/Size/Apparel Curated content

Ease of Use Am I able to complete the scenario? Is it complicated? Do I need to perform extraordinary steps to get what I need done? Are there glitches in the system that make it difficult? Was it hard to find how to execute this scenario? Are the features hidden? Is the experience consistent?

Responsiveness Did I feel like the scenario was fast and fluid? At any point did I feel like I had to wait a frustrating amount of time? Was I effectively distracted while waiting for an action to take place? (ex. Movie or animation while I wait)

Usefulness Would you use this scenario yourself to accomplish this specific outcome? Does this scenario meet a need for our consumer? Does this let me do something that I want to do?

Visual Appeal Is the experience exciting to see and hear? Is the UI polished? Does the UX make the experience enjoyable?

Evaluation Scale 5 Love It! – You love it so much you’d shout it out at the top of your lungs through a bullhorn from every rooftop you encounter 4 Like It – You like it a lot and might mention it during a lull in conversation at a dinner party 3 Meh – You can live with it and neither like it or hate it. It’s nothing special. There are some improvements that can be made 2 Don’t Like It – The experience leaves a bad taste in your mouth. You’d use it if you really had to otherwise, you’d stay away from it. 1 Hate it! – You hate it so much that you would only use it if you were under a Hogwarts compulsion spell 0 Not Implemented

ScenarioLove ItLike ItMehDon't LikeHate Do this Do that Do the other thing Do something else

Controlled “A/B” Experiments Concept: Randomly split traffic between two (or more) versions A (Control) B (Treatment) Collect metrics of interest Analyze

Let’s play a game Three choices are: A wins B wins A and B are approximately the same (*) results based on statistically significant data

MSN Real Estate “Find a house” widget variations Revenue to Microsoft generated every time a user clicks search/find button Raise your left hand if you think A Wins Raise your right hand if you think B Wins Don’t raise your hand if you think they’re about the same A B

MSN Real Estate If you did not raise a hand, please sit down If you raised your right hand, please sit down A was 8.5% better

Office Online Test new design for Office Online homepage A Clicks on revenue generating links (red below) Raise your left hand if you think A Wins Raise your right hand if you think B Wins Don’t raise your hand if you think they’re about the same B

Office Online If you did not raise a hand, please sit down If you raised your right hand, please sit down A was 64% better The Office Online team wrote A/B testing is a fundamental and critical Web services… consistent use of A/B testing could save the company millions of dollars

MSN Home Page Search Box Click-through rate for Search box and popular searches A B Differences: A has taller search box (overall size is the same), has magnifying glass icon, “popular searches” B has big search button Raise your left hand if you think A Wins Raise your right hand if you think B Wins Don’t raise your hand if they are the about the same

Search Box Search Box If you raised any hand, please sit down Don’t change for the sake of change Sometimes what you have is enough 24

Tester Developer Designer

Math Geek Developer Designer

Choose your test / experiment Measure the results Determine what to do next

BuildMeasureLearn

“Given the baseline conversion rate on opens, the sample size simply isn’t large enough to get a reliable result.”

Tracing (Build) Collect Logs & upload (Measure) Analyze (Learn) startTime = GetTime()); CoolActivity(); Trace(ID, ACTIVITY_LENGTH, GetTime() - startTime)

Is your product ready to ship?

Does your product satisfy your internal quality criteria? Is your product ready for customer feedback? Is your team ready for customer feedback?

There are no Silver Bullets

Customer data can help Use both qualitative and quantitative data Customer data can replace some testing Data does not replace excellent human testing