User Research HCDE 518 & INDE 545 Winter 2012 With credit to Jake Wobbrock, Dave Hendry, Andy Ko, Jennifer Turns, & Mark Zachry

Agenda Announcements Lecture & Discussion – Design Methods Break – 5 mins Lecture & Discussion – CI & Ethnography Next class

Announcements P0, R1, R2 graded This Wednesday – no class meeting A1 shortly This Wednesday – no class meeting Group work day A2 deadline extended to next Monday Next week is normal (I hope!)

Design Methods

Observing People What do we “see”? Opportunities for new designs Breakdowns Workarounds Mismatches between what users say and do

Relying on what users say Can we rely on what users say about what they want in a new design? Very carefully Henry Ford: “If I had asked my customers what they wanted, they would have said a faster horse." It is better to watch what they do than to go only on what they say Mismatches may hold keys to new designs

Users’ words are unreliable People are notoriously bad at predicting what they would use or would prefer when it is only hypothetical They can much better respond to actual, concrete things, or make comparisons This highlights the importance of observation and of prototypes

Users can however… Tell you what they are doing right now Tell you how they are feeling right now Tell you what their goal is right now

IDEO Method Cards Available from William Stout publishers ($49)

LEARN from the facts you gather

LOOK at what users really do

ASK users to help

TRY it yourself

Triangulation All user research techniques have their own limitations Use multiple techniques to fully understand a design scenario Choose techniques that account for the weaknesses of each other Choose techniques to cover both a breadth and depth of the user experience

Design Activity – 10 minutes Using the Method Cards, come up with two methods that could be useful in each of the following contexts, and two that would not be useful for the following design question. Make sure they triangulate! How can cell phones help older adults communicate with their young grandchildren over a distance? Cards: http://courses.washington.edu/hcde518/readings/IDEOMethodCards.pdf http://courses.washington.edu/hcde518/readings/IDEOMethodCards.pptx

What did you come up with?

A2 – Look, Learn, Ask, Try Similar to what you just did! You’ll be given 3 design scenarios and you’ll be asked to come up with 4 methods that would be appropriate and 1 that would not How can a new system support communication for emergency room nurses? How can a mobile system help long-distance bicyclists to find restaurants and amenities? How can a video game help educate kids in Grades 1-5 on healthy eating You’ll be asked to explain your choices Due next Monday (1/30)

Diana’s Discussion Questions 1.     Is it better to start with looking, asking, learning or trying? 2.     Which method(s) from the 51 cards best addresses the appearance of an artifact? 3.     Which method(s) from the 51 cards best addresses the culture of a population? 4.     What method(s) is less time consuming? In general, how much time does a designer needs to save for this phase? 5.     Methods like surveys are easy to mass distribute, whereas interviews are more difficult to obtain. How meaningful is the information from one another?

Diana’s Discussion Questions 6.     If design shapes human experience, why not let the user come up with the first prototype? Wouldn’t that say more about how they perceive the product? 7.     Goals like organic work flow cannot always be achieved. From the four orders of design, what should take priority? 8.     When does the designer develop the working hypothesis? (That is, what kind of research is originally done?) 9.      How does thinking of technology as systematic thinking differ to thinking of it as products? 10.  Most of the reading seem to describe ideal processes for knowing people and designing, why are they not followed?

Break – 5 minutes

P1 – Due Monday, Feb. 8th Define stakeholders Choose and execute 3 user research techniques (make sure they complement each other) Cannot be from the “Try” category Define design requirements Create personas Re-visit design question

Ethnography Observational science attempts to understand a group or individual objectively Understand the subject of study from the outside in a way that can be explained to “anyone” Generate “thick description” painting a vivid holistic picture Ethnography attempts to understand a group or individual phenomenologically Understand the subject of study as the subject of study understands himself/herself

Design Ethnography Quicker than traditional ethnography Usually days, weeks, or months, not years Sometimes called “concurrent ethnography” The ethnography is being done at the same time that design is under way Goal is to generate insights for informing inspiring design Translating from raw field data to design ideas can be difficult

Four Ethnographic Principles

Natural Settings Conducted in the setting of the participant Focus on naturally occurring, everyday talk and action Cannot use laboratory or experimental settings to gather this type of data

Holism Behavior can only be understood in its larger social context That is, holistically

Descriptive Study how people actually behave, not how they ought to behave Defer judgment Data is not usually quantitative, but qualitative

Subjects' Point-of-View See through participants' eyes in order to grasp how they interpret and act in their world Phenomenological

Personal Example Studying autism therapists Trained to be a therapist, acted as one for 6 months Immensely helpful in gaining trust and access to users Could easily test out prototypes

Contextual Inquiry What does context refer to?

Key Concepts “The users cannot describe what they really do because they are not conscious of it and do not reflect on it.” “The core premise of Contextual Inquiry is very simple: go where the customer works, observe the customer as he or she works, and talk to the customer about the work. Do that, and you can’t help but gain a better understanding of your customer.”

Principles of Contextual Inquiry Must be done in the setting of the participant Partnership Master/apprentice model; investigator is humble Interpretation Observed facts must be regarded for their design implications. Raw facts without interpretation aren't very useful Focus Themes that emerge during the inquiry. You can't pay attention to all facets of someone's work at all times!

Master/Apprentice You are the apprentice The informant is your master What does this relationship imply? Keen observation Unafraid to ask questions Eager to learn Admire the master Aspire to see the world as they do Adopting the master/apprentice model during your CI will mean you don't have to pre-prepare a set of interview questions Reduces pressure to “get it right.”

Interviewing in CI Go for concrete details obtained in-context, not abstract generalities Don't ask participants to summarize their work Ask them specific details about real, concrete, observable things Have them “think aloud” as they work through their tasks Pepper them with short, easily answerable questions Avoid high-level philosophical questions that will just cause them to “talk” instead of “do”

Interpretation Checking It is good to regularly check your interpretations “I saw you just do X. Is that because of Y?” “I believe X. Is that correct?” “If you had a technology that did X, would that solve the problem we just encountered?” As long as you check your interpretations in context, participants will respond accurately Outside of context, they may be more inclined to agree or answer in generalities rather than specifics

Ways to Mess Up a CI Not being inquisitive/nosy enough If you have the impulse to ask, do it right away! Overly disrupting the task Questions are great, but don’t ask so many so fast that the participant stops doing their tasks. Turning it into a regular interview If you could have done it in a coffee shop, you didn’t do a contextual inquiry. Failing to be discrete Participants must feel safe, free, and anonymous. Failing to respect your participants Failing to observe closely and take good notes Over-focusing on the wrong details Slipping into abstraction Keep it concrete, in the work, in the details.

Summary The Design Cards, Design Ethnography, and CI are all ways of understanding your users’ goals and needs better Focus on: Understanding what they’re trying to accomplish How they do it now What works What doesn’t work -> Design opportunity! More info -> See backup slides

Siri’s Questions https://catalyst.uw.edu/gopost/conversation/jkientz/599293

Case Studies Huang & Truong Tee, Brush, & Inkpen Questionnaire and Interviews Focus on understanding how and why people dispose of their old mobile phones Identify design opportunities Tee, Brush, & Inkpen Interviews, family trees Understand ways that families currently communicate

Next Class Topics Wednesday, January 25th Monday, January 30th No class lecture – group work time Monday, January 30th Personas Discussant - Patrick Upcoming Work Get started on P1 user research! A2, R4

Group Project time

backup

Data Recording Approaches Notes Notes + still camera Notes + Audio Notes + Audio + still camera Video What are the advantages and disadvantages to each of these?

Interviews Purpose: Collect detailed information about tasks, activities, technologies. Understand the why behind activities, rather than the what. Suitable for relatively small number of people (5 – 30) - Shoot for ~12 to reach data saturation, though not always feasible

Interviews Unstructured Broad questions concerning some general area Structured Narrow questions concerning specific area Semi-structured Balance between broad/narrow questions Focus groups Group discussion around a topic

Creating an Interview Guide Who do you need to interview and why? Demographic questions Open questions Closed questions Activities Sketching Demonstrations Reliability and validity Be careful about leading questions

Running an Interview 0. Recruitment Introduction Warm up session Main session Cool-off period Closing session

Example – Huang & Truong https://courses.washington.edu/hcde518/readings/MobilePhoneInterviewProtocol.pdf

Questionnaires/Surveys Purpose: Deepen understanding by collecting information from a broad range of people Suitable for large number of people 20 – 1,000+

Questionnaires Be clear on the goal Keep it short Open and closed questions Rating scales (e.g. Likert) Be sure to pilot your questionnaire and expect to iterate 3-4 times Online or on paper?

Example – Huang & Truong https://courses.washington.edu/hcde518/readings/MobilePhoneSurvey.pdf

Observations in the Field Helps you understand people’s context, task, activities, goals What people do is different than what people say they do People are notoriously poor and describing their work

What to Pay Attention To Key features Questions Space Actors Activities Objects / workarounds Acts Events / triggers Time Goals Feelings How is the physical space adapted to the job? What are the key constraints on the job? Where are strategic decisions made? Where are tactical decisions made?

Examples What might you observe about each case? Is observation appropriate? A nurse as she treats patients in a hospital A teacher while he is teaching a classroom full of kindergartners A team of emergency medical technicians working on a case A writer working on a book

Laboratory Studies Useful for studying and recording the details of how people perform: Goals Tasks Action sequences Disadvantages: Not the user’s natural environment, missing context Better for doing tests of prototypes where detail is important

Lab Studies

Indirect Observations Diary studies Experience Sampling studies Interaction logs

Diary Studies Have users carry a diary with them to answer specific questions about activities throughout the day when certain activities occur

Example PAL diary study Fits in one’s pocket Record when PAL is desirable

Experience Sampling Studies Have users carry around a device that has them answer questions at given intervals Cell phone, PDA, SMS Good for getting in-the-moment assessments Example: page user once every 3 hours and ask them to fill out a short survey on their current activity and rate sleepiness level on a scale from 1 to 7

Example – Track Your Happiness Go to TrackYourHappiness.org and sign up

Diary Study vs. Experience Sampling Both can collect similar types of information Diary is less intrusive ES can be more reliable at getting regular data User is less likely to forget Another alternative – context-based sampling MyExperience Toolkit - http://myexperience.sourceforge.net/

Interaction Logs / Usage Logs Use software to automatically log interactions with a system e.g., number of clicks, time spent on task, etc. e.g., pages visited within a site: Google Analytics Can also be used on other platforms using interesting sensors to sense context of use Example: Logging steps with a pedometer

Example Using Bluetooth tags to track people’s distance from their cell phones Intent: Inform design of “always on” ubiquitous computing applications

Case Studies Huang & Truong Tee, Brush, & Inkpen Questionnaire and Interviews Focus on understanding how and why people dispose of their old mobile phones Identify design opportunities Tee, Brush, & Inkpen Interviews, family trees Understand ways that families currently communicate

Questions What were your overall thoughts? What did you like about the approach? What would you have changed about the approach? Why do you think they chose the approach they did?