1. Observing users Presenter : JungMin shin & JaeYoung Lee

2. Contents Goals, questions, paradigms Goals, questions, paradigms How to observe How to observe Data Collection Data Collection Indirect Observation: tracking user’s activities Indirect Observation: tracking user’s activities Analyzing, interpreting, and presenting data Analyzing, interpreting, and presenting data Field studies Field studies

3. Goals, questions, paradigms What and when to observe What and when to observe Approached to observation Approached to observation

4. Goals, questions, paradigms Goal and question provide a focus for observation, DECIDE framework. Having goal helps to guide the observation because there is always so much going on. What and when to observe What and when to observe On developing product On developing product starting time : helps understand user’s needs. ending time : examine to satisfy user’s needs.

5. Goals, questions, paradigms Role of evaluator Role of evaluator Onlookers, participant observers, ethnographers Table 12.1 Type of observation Observation Controlled environment (i.e., lab-like) Field environment (i.e., natural) Outsider looking on “Quick and dirty” In usability testing “Quick and dirty” In field studies Insider (Not applicable) Participant observation (e.g., ethnography)

6. Goals, questions, paradigms Approaches to observation Approaches to observation 1. “Quick and dirty” observation Watching and talking to users in a casual way to get immediate feedback. 2. Observation in usability testing Video and interacting logs capture everything. Can watch through a one-way mirror or via TV screen. Observational data is used to see and analyze for user’s action.

7. Observation in usability testing System Observation in usability testing System Goals, questions, paradigms

8. 3. Observation in field studies Observers may be anywhere along the outsider – insider spectrum. Colin Robson(1993) – participating level complete participants more marginal participants observers who also participate observers who also participate people who observe from the outside and do not participate. The goal is to cause as little disruption as possible.

9. Goals, questions, paradigms 3. Observation in field studies (Continue) ex) ex) A study of the time spent by boy and girl using technology in the classroom. -> observe and note at the back of room. A study of understanding how the computer integrates with other artifacts and social interacting. -> might take of insider perspective.

10. Goals, questions, paradigms 3. Observation in field studies (Continue) participant observation evaluator participant observation evaluator participate with users in order to learn what they do and how and why they do it. fully participant observer fully participant observer a member of group. being present to share experience and learning the social conventions of the group.

11. Goals, questions, paradigms Determining goals, exploring questions Choosing techniques are necessary steps in the DECIDE framework.

12. In controlled environments In controlled environments In the field environments In the field environments Participant observation and ethnography Participant observation and ethnography How to observe

13. In controlled environments In controlled environments Collect and then Make sense of the data while watching users in controlled environment. Practical issues in controlled environment - where users will be located - equipment test - An informed consent form How to observe

14. In controlled environments In controlled environments the observer doesn’t know what users are thinking, and can only guess from what she sees. Think-aloud technique by Erikson and Simon Require people to say out loud everything that they are thinking and trying to do. ex) I’m typing in http://ics.uci.edu (type) http://ics.uci.edu Now I press the enter key, right? (presses enter key) Now I press the enter key, right? (presses enter key) But if users meet some errors, they could be silent again... Sol) Evaluator reminds him think out loud, but that would be intrusive. -> two people work together so that they talk to each other. How to observe

15. In the field environments In the field environments Events in the field can be complex and rapidly changing. Framework help observers to keep their goals and questions in sight. Collin Robson(1993) Space, Actors, Activities, Objects, Acts, Events, Goals, Feelings How to observe

16. In the field environments In the field environments Checklist of things to plan before going into field Checklist of things to plan before going into field 1. State the initial study goal and question 2. Select the framework 3. Decide how to record events 4. Be prepared to go through your notes and other records 5. Make and review your notes 6. Be prepared refocus your study 7. Think about how you will get the acceptance and trust of those you observe those you observe 8. Think about how to handle sensitive issues (privacy) 9. Consider working as a team 10. Consider checking your notes with an informant or members of the group of the group 11. Plan to look at the situation from different perspectives How to observe

17. Participant observation and ethnography Participant observation and ethnography Checklist for doing ethnography Checklist for doing ethnography Identifying a problems or goal and then ask good questions to be answered by the study Identifying a problems or goal and then ask good questions to be answered by the study The most important part of fieldwork just being there to observe, ask questions, and record. The most important part of fieldwork just being there to observe, ask questions, and record. Collect a variety of data (Interview, retrospectives interview) Collect a variety of data (Interview, retrospectives interview) Be prepared to move backwards and forwards between the broad picture and specific questions Be prepared to move backwards and forwards between the broad picture and specific questions Analyze the data using a holistic approach -> contextualized, usually iterative Analyze the data using a holistic approach -> contextualized, usually iterative How to observe

18. Participant observation and ethnography Participant observation and ethnography When have I observed enough? schedules often dictate when your study end. stop when you stop learning new things when you start to see similar pattern of behavior being repeated. when you have listened to all the main stake holder group and understood their perspectives. How to observe

19. Participant observation and ethnography Participant observation and ethnography How can I adapt ethnography to fit the development process? Ann Rosa Procedure Preparing (Understand/Familiarize/Set Goal /Gain permission) permission) Field study (Establish/Observe and interview/ Follow any lead/ Record) Follow any lead/ Record) Analysis (Compile/Quantify/Reduce and interpret/ Refine the goals) Reporting (Consider/Prepare)

20. Data collection Notes plus still camera Notes plus still camera Audio recording plus still camera Audio recording plus still camera Video Video

21. Data collection Notes plus still camera Notes plus still camera being difficult and tiring to write and observe at the same time bored, the limitation of writing speed (sol) working with another person (sol) working with another person (disadvantage) have to be transcribed Audio recording plus still camera Audio recording plus still camera Useful alternative to note taking and is less intrusive than Video. Drawback: Lack of visual record, transcribing the data So, evaluators use the recording to remind them about important details and as a source of anecdotes for reports.

22. Data collection Video VideoAdvantage Capturing both Audio and Video Data DisadvantageIntrusive. Attention becomes focused on what is seen through the lens. Time-consuming for analyzing video data. ex) 1 hour video recording : over 100 hours of ex) 1 hour video recording : over 100 hours of analysis time analysis time

23. Indirect Observation: tracking user’s activities Diaries Diaries Interaction logs Interaction logs

24. Indirect Observation: tracking user’s activities When ? Impossible to observe directly Diaries Diaries A record of what users did, when did it, what thought about their interactions with the Technology. (Advantage) When User scattered and unreachable person. Inexpensive - no equipment or expertise. Suitable for long term studies. Template and Enable the data to go straight into database. Ex) open-ended online questionnaires (Disadvantage) Rely on participant (reliable/remembering). Participant remember better or worse that they really were.

25. Interaction logs Interaction logs key presses, mouse or other device movement are recorded usually synchronized with A/V log. Has time-stamped (to calculate how long) Explicit counter / Recording number of visitors EX) In study of interact art museum (Researchers at USC) Server log (7 months/ analysis tool : webtrends) when/what /how long/what browser/what country… (Advantage) (Advantage) Unobtrusive (but this also raises ethical concerns) Large volumes of data. Indirect Observation: tracking user’s activities

26. Analyzing, interpreting, and presenting data The first things to do is to “eyeball”. Proceed to analyze it according to the goals and questions. Qualitative analysis to tell a story Qualitative analysis to tell a story Qualitative analysis for categorization Qualitative analysis for categorization Quantitative data analysis Quantitative data analysis Feeding the findings back into design Feeding the findings back into design

27. Qualitative analysis to tell a story Qualitative analysis to tell a story Power of analyzing descriptive data -> being able to tell a convincing story Interview with Sara Bly (User-Centered Design Consultant) Interview with Sara Bly (User-Centered Design Consultant) At the end of each observation period Review their data Review their data Discuss what they observed Discuss what they observed Construct a story from their data Construct a story from their data Analyzing, interpreting, and presenting data

28. Qualitative analysis to tell a story Qualitative analysis to tell a story To summarize about Interview with Sara Bly. Review the data. Record the themes. Record the date and time of each data analysis Session. As themes emerge. Iterate this process (until your story faithfully represents). Report your findings to the development team. Analyzing, interpreting, and presenting data

29. Qualitative analysis to tell a story Qualitative analysis to tell a story Analyzing and reporting ethnographic data Analyzing and reporting ethnographic data - ethnographers reconstruct knowledge to produce detailed description known as rich or thick description. Main activities (Fetterman-1998) Main activities (Fetterman-1998) Look for key events Look for key events Look for patterns of behavior in various situations and among different players Look for patterns of behavior in various situations and among different players Compare sources Compare sources Report your findings in convincing and honest way Report your findings in convincing and honest way S/W tools : NUDIST and Ethnograph S/W tools : NUDIST and Ethnograph Analyzing, interpreting, and presenting data

30. Qualitative analysis for categorization Qualitative analysis for categorization Looking for incident or patterns Common strategy is to look for critical incidents Common strategy is to look for critical incidents ex) Jurgen Koenemann-Belliveau el al(1994) used this form of critical incident analysis to examine breakdown or problems in achieving a programming task and also to identify possible threats of incidents. Using the theory helped the evaluators to focus in relevant Incidents. Analyzing, interpreting, and presenting data

31. Qualitative analysis for categorization Qualitative analysis for categorization analyzing data into categories contents analysis provides another fine grain way of analyzing video data. determined by the evaluation question. must also be reliable so that the analysis can be replicated. EX) training second person and then analyze the same data sample. if there are large discrepancy between each other. -> what? Inter-research reliability rating -> the percentage of agreement between the two researchers. Analyzing, interpreting, and presenting data

32. Analyzing discourse Analyzing discourseDialog Strongly interpretive, pays great attention to context. Language is viewed as a constructive tool and discourse analysis provides a way of focusing upon how people use language to construct versions of their worlds (Fiske,1994) Small changes in wording can change meaning, as the following excerpts indicate (Coyle, 1995) (According to Coyle,) discourse analysis is what you do when you are saying that you are doing discourse analysis… Analyzing, interpreting, and presenting data

33. Quantitative data analysis Quantitative data analysis In case of errors or unusual behavior -> marks the video and records a brief remark. marks the video and records a brief remark. To use this annotated recording 1. Calculate performance time 1. Calculate performance time 2. further analyzed using simple statistics such as means, standard deviations, T- tests, etc. Analyzing, interpreting, and presenting data

34. Feeding the findings back into design Feeding the findings back into design The results can be reported to the design team. Written reports with a overview and detailed content list. Quantitative data -> its value depends on the type of study and its goals. Often both quantitative data analysis are useful because of they provide alternative perspectives. Analyzing, interpreting, and presenting data

35. Field Study

36. A broad range of data gathering techniques at the user’s location A broad range of data gathering techniques at the user’s location There is no one best way to conduct a field study. It depends on the goals of your study and your access to users. There is no one best way to conduct a field study. It depends on the goals of your study and your access to users.

37. The Goal of a Field Study Field studies are excellent for collecting rich, detailed data and obtaining a holistic view of the process or domain. Field studies are excellent for collecting rich, detailed data and obtaining a holistic view of the process or domain. Field studies are not a good technique if you want to collect quantitative data from large samples. Field studies are not a good technique if you want to collect quantitative data from large samples. The goal of a field study is to make the implicit aspects and processes of the user environment explicit. The goal of a field study is to make the implicit aspects and processes of the user environment explicit.

38. Field Study Methods Techniques are divided into three categories: Techniques are divided into three categories: Observation only Observation only Interacting with the user Interacting with the user Method supplements Method supplements The Goal of the Field Study Methods The Goal of the Field Study Methods To observe users and collect information about their tasks and the context in which they are done. To observe users and collect information about their tasks and the context in which they are done.

39. Field Study Methods Observation Only Observation Only Pure Observation Pure Observation Deep Hanging-Out Deep Hanging-Out Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO) Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

40. Field Study Methods Observation Only Observation Only Pure Observation Pure Observation Deep Hanging-Out Deep Hanging-Out

41. Observation only - Pure Observation Synopsis Synopsis When you are unable or don’t wish to interact with the user, you simply observe from a distance. When you are unable or don’t wish to interact with the user, you simply observe from a distance. Advantages Advantages Flexible Flexible Low resources Low resources Level of effort Level of effort Minimal, Place your self in a good vantage point and observe as many user/sites/tasks as you feel appropriate Minimal, Place your self in a good vantage point and observe as many user/sites/tasks as you feel appropriate You continue to conduct observations until you feel you have a good understanding of the domain of areas of focus. You continue to conduct observations until you feel you have a good understanding of the domain of areas of focus.

42. Observation only - Pure Observation Valuable in situations where you cannot interact with the end users. Valuable in situations where you cannot interact with the end users. Users may or may not know they are being studied. Users may or may not know they are being studied. Do not interact with the participant Do not interact with the participant Do not distribute surveys, interview the user, or ask for artifacts from the user. Do not distribute surveys, interview the user, or ask for artifacts from the user. It is essential to have a good sampling plan. The sampling plan should include specific information such as days/times you anticipate key events (e.g., the day before Thanksgiving, or bad weather at an airport), as well as “normal” days. It is essential to have a good sampling plan. The sampling plan should include specific information such as days/times you anticipate key events (e.g., the day before Thanksgiving, or bad weather at an airport), as well as “normal” days.

43. Field Study Methods Observation Only Observation Only Pure Observation Pure Observation Deep Hanging-Out Deep Hanging-Out

44. Observation only – Deep Hanging out Synopsis Synopsis This method is similar to pure observation but provides more structure by suggesting focus areas and things to observe. This method is similar to pure observation but provides more structure by suggesting focus areas and things to observe. Advantages Advantages It has more structured than pure observation so you can do a more detailed level of data analysis and compare data collected across multiple sites. It has more structured than pure observation so you can do a more detailed level of data analysis and compare data collected across multiple sites. Level of effort Level of effort Because there is more structure, it takes more effort than pure observation. Because there is more structure, it takes more effort than pure observation. You are “on” at all times, which can be tiring. You are “on” at all times, which can be tiring. It is also valuable to become a user yourself (if possible) and collect artifacts along the way. It is also valuable to become a user yourself (if possible) and collect artifacts along the way.

45. Observation only – Deep Hanging out A more structured form of pure observation. A more structured form of pure observation. Involves significant amounts of observation along with involving yourself in the process Involves significant amounts of observation along with involving yourself in the process Method of Deep Hanging-Out includes Method of Deep Hanging-Out includes Structured observation Structured observation Collection of artifacts Collection of artifacts Becoming a user yourself Becoming a user yourself However, you do not interview participants, distribute surveys, or present design ideas for feed back. However, you do not interview participants, distribute surveys, or present design ideas for feed back.

46. Field Study Methods Observation Only Observation Only Pure Observation Pure Observation Deep Hanging-Out Deep Hanging-Out Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO) Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

47. Field Study Methods Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO)

48. Interacting with the user - Contextual Inquiry Synopsis Synopsis Interview, apprentice with, and interpret the resulting data with users Interview, apprentice with, and interpret the resulting data with users Advantages Advantages Contextual Inquiry is more focused and context dependent than the other methods Contextual Inquiry is more focused and context dependent than the other methods At the end, you walk away with actionable items At the end, you walk away with actionable items Level of effort Level of effort The effort level is higher than for observation-only techniques. You must develop an observation guide, observe users, apprentice with them, and discuss your observations with them The effort level is higher than for observation-only techniques. You must develop an observation guide, observe users, apprentice with them, and discuss your observations with them

49. Interacting with the user - Contextual Inquiry Four main parts to Contextual Inquiry: Four main parts to Contextual Inquiry: Context Context Partnership Partnership Interpretation Interpretation Focus Focus

50. Interacting with the user - Contextual Inquiry Four main parts to Contextual Inquiry: Four main parts to Contextual Inquiry: Context : You must go to the user’s environment in order to understand the context of his/her actions. Contextual Inquiry assumes that observation alone or out-of- context interviews are insufficient Context : You must go to the user’s environment in order to understand the context of his/her actions. Contextual Inquiry assumes that observation alone or out-of- context interviews are insufficient

51. Interacting with the user - Contextual Inquiry Four main parts to Contextual Inquiry: Four main parts to Contextual Inquiry: Partnership : To better understand the user, tasks, and environment, you should develop a master-prentice relationship with the participant. Immerse your self in the participant’s work and do as he/she does. Obviously, this is not possible with many jobs (e.g., surgeon, fighter pilot) Partnership : To better understand the user, tasks, and environment, you should develop a master-prentice relationship with the participant. Immerse your self in the participant’s work and do as he/she does. Obviously, this is not possible with many jobs (e.g., surgeon, fighter pilot)

52. Interacting with the user - Contextual Inquiry Four main parts to Contextual Inquiry: Four main parts to Contextual Inquiry: Interpretation : Observations must be interpreted with the participant in order to be used later. Verify that your assumptions and conclusions are correct. Interpretation : Observations must be interpreted with the participant in order to be used later. Verify that your assumptions and conclusions are correct.

53. Interacting with the user - Contextual Inquiry Four main parts to Contextual Inquiry: Four main parts to Contextual Inquiry: Focus : Develop an observation guide to keep you focused on the subject of interest/inquiry. Focus : Develop an observation guide to keep you focused on the subject of interest/inquiry.

54. Field Study Methods Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO)

55. Interacting with the user – Process Analysis Synopsis Synopsis Capture the task sequence for a process that may span several days Capture the task sequence for a process that may span several days Advantages Advantages Because it is more focused than Contextual Inquiry, it is also much faster Because it is more focused than Contextual Inquiry, it is also much faster Level of effort Level of effort You need to stay focused on the process at hand to help users walk you through the process of interest. It may span just several days. You need to stay focused on the process at hand to help users walk you through the process of interest. It may span just several days.

56. Interacting with the user – Process Analysis A process analysis is similar to Contextual Inquiry. However, you enter with a series of questions and you do not necessarily apprentice with the user A process analysis is similar to Contextual Inquiry. However, you enter with a series of questions and you do not necessarily apprentice with the user Questions Questions When does the first task in the process happen? When does the first task in the process happen? What triggers it? What triggers it? Who does it? Who does it? What information does the person have when the task begins? What information does the person have when the task begins? What are the major steps in the task? What are the major steps in the task? What information comes out of it? What information comes out of it? Who is the next person in the chain of the process? Who is the next person in the chain of the process?

57. Interacting with the user – Process Analysis Example - Process map for a travel agent’s typical day Example - Process map for a travel agent’s typical day

58. Field Study Methods Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO)

59. Interacting with the user – Condensed Ethnographic Interview Synopsis Synopsis Use the results of semi-structured interviews to guide observations Use the results of semi-structured interviews to guide observations Advantages Advantages This technique is considerably shorter than some of the other techniques, because the interviews scope what you observe; This technique is considerably shorter than some of the other techniques, because the interviews scope what you observe; It also limits the data you are able to collect It also limits the data you are able to collect Level of effort Level of effort It requires a medium level of effort to conduct interviews and then observe users and collect artifacts It requires a medium level of effort to conduct interviews and then observe users and collect artifacts

60. Interacting with the user – Condensed Ethnographic Interview Employs the standardization and focus of a semi-structured interview along with the context of observations and artifacts Employs the standardization and focus of a semi-structured interview along with the context of observations and artifacts Characterized as “top-down”, because the interviews form a general framework from which to interpret specific observations. Characterized as “top-down”, because the interviews form a general framework from which to interpret specific observations.

61. Interacting with the user – Discount User Observation (DUO) Synopsis Synopsis One facilitator interviews the user and takes notes while another facilitator takes pictures of everything One facilitator interviews the user and takes notes while another facilitator takes pictures of everything Advantages Advantages You can quickly reconstruct the session through time stamped notes and digital pictures to verify your observations with users You can quickly reconstruct the session through time stamped notes and digital pictures to verify your observations with users Level of effort Level of effort It takes two investigators to focus on their specific tasks of either interviewing or taking pictures and notes It takes two investigators to focus on their specific tasks of either interviewing or taking pictures and notes It will take additional time/effort to combine notes with digital photos and put them in a multimedia presentation tools to review with users. It will take additional time/effort to combine notes with digital photos and put them in a multimedia presentation tools to review with users. It takes additional effort to validate observations with user afterward. It takes additional effort to validate observations with user afterward.

62. Field Study Methods Observation Only Observation Only Pure Observation Pure Observation Deep Hanging-Out Deep Hanging-Out Interacting with the user Interacting with the user Contextual Inquiry Contextual Inquiry Process analysis Process analysis Condensed ethnographic interview Condensed ethnographic interview Discount User Observation (DUO) Discount User Observation (DUO) Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

63. Field Study Methods Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

64. Method supplements – Artifact walkthroughs Synopsis Synopsis Collect all the artifacts used by participants and determine what triggers their use, when they are used, and for what Collect all the artifacts used by participants and determine what triggers their use, when they are used, and for what Advantages Advantages Quick and easy to conduct Quick and easy to conduct Level of effort Level of effort Low level of effort to review artifacts with participants and make/collect copies of them Low level of effort to review artifacts with participants and make/collect copies of them

65. Method supplements – Artifact walkthroughs Quick and easy to conduct but provide indispensable data Quick and easy to conduct but provide indispensable data “Artifacts” are objects or items that users use to complete their tasks or that result from their tasks “Artifacts” are objects or items that users use to complete their tasks or that result from their tasks You have to sign the company’s confidential disclosure agreement promising that you will keep all data collected confidential You have to sign the company’s confidential disclosure agreement promising that you will keep all data collected confidential

66. Field Study Methods Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

67. Method supplements – Incident diaries Synopsis Synopsis Worksheets the user takes home or to work to collect ongoing data Worksheets the user takes home or to work to collect ongoing data Advantages Advantages No observation is required No observation is required Able to understand more issues than what can be observed in the lab or during a single visit Able to understand more issues than what can be observed in the lab or during a single visit Level of effort Level of effort Low effort level to create and distribute diaries Low effort level to create and distribute diaries Moderate effort to analyze data across multiple diaries. There will be a time lag between when you distribute the diaries and when you receive the data Moderate effort to analyze data across multiple diaries. There will be a time lag between when you distribute the diaries and when you receive the data You are depending on the participants to follow-thorough. You are depending on the participants to follow-thorough.

68. Method supplements – Incident diaries Incident diaries allow you to collect ongoing data from users Incident diaries allow you to collect ongoing data from users Participants are provided with a notebook containing worksheets to be completed Participants are provided with a notebook containing worksheets to be completed They are given to users to keep track of issues they encounter while using a product They are given to users to keep track of issues they encounter while using a product The worksheets may ask users The worksheets may ask users Describe a problem or issue they encountered Describe a problem or issue they encountered How they solved it How they solved it How troublesome it was How troublesome it was

69. Field Study Methods Method supplements Method supplements Artifact walkthroughs Artifact walkthroughs Incident diaries Incident diaries Observing while you are not present Observing while you are not present

70. Method supplements – Observing while you are not present Synopsis Synopsis Recording users in action when space, time, or restrictions prevent you from being there in person Recording users in action when space, time, or restrictions prevent you from being there in person Advantages Advantages If you have multiple video cameras, you can view several users simultaneously If you have multiple video cameras, you can view several users simultaneously Level of effort Level of effort Low effort to set up cameras and record Low effort to set up cameras and record Moderate effort to meet with user again and review tapes Moderate effort to meet with user again and review tapes Moderate effort to categorize and index behavior Moderate effort to categorize and index behavior

71. Method supplements – Observing while you are not present You can observe users even when you are not present by setting up a video camera and then leaving You can observe users even when you are not present by setting up a video camera and then leaving Excellent way to understand detailed steps Excellent way to understand detailed steps

72. Identify the type of study to Conduct decision diagram decision diagram

73. Thank you~