1. Diaries KSE966/986 Seminar - Fall 2012/Spring 2013 March 29, 2013 Uichin Lee

2. Contents Why use diaries in HCI? Challenges with diaries Participants for a diary study Types of Diaries (feedback, elicitation) Data collection When to record an entry? Analysis of diaries

3. What is a diary? A diary is a document created by an individual who maintains regular recordings about events in their life, at the time that those events occur Diaries are useful for recording information that is fluid and changes over time ◦ User mood, feelings, such as frustration, anger, and pain, (mobile) information needs (any types of multiple events occurred within a day) If recall after-the-fact will be hard, inaccurate, or subject to biases, diaries can be a useful data collection method

4. What is a diary? Informal diaries: ◦ Blogs, status updates using Facebook, or “tweets” on Twitter ◦ Entries are made as they occur, in real-time (although there are no stated research purposes) Diaries have been adopted, from sociology and history, for use in human- computer interaction research

5. What is a diary? Example: time diary ◦ A time diary is when there is a focus on how time is used, and entries need to be made on a regular basis (with a specified dimension of time) ◦ Time diaries are useful, because people often remember time inaccurately Diary is a useful approach as much of research in HCI focuses on understanding how we interact software and hardware artifacts in our daily lives

6. Why use diaries in HCI? Diaries fill the gap between observation in naturalistic settings and fixed laboratory settings, and surveys Users may have different reactions when being observed, and observers may not always understand what is going on If interested in collecting data that is fluid and changes over time (rather than factual data), surveys can lead to biased data due to biases in recall Multi-method research is often the best approach ◦ as all methods have strengths and weaknesses

7. Why use diaries in HCI? Diaries allow for more collecting more detailed research data than surveys Survey has predefined questions w/ little flexibility given to the respondents ◦ Mostly appropriate if we ask users to recall info that does not change over time Diaries are good at understanding not only what users are doing, but why they are doing it Dairies are useful for collecting: ◦ Fluid data: occurring only at a specific time, and changing, e.g., mood, feelings, perception, time, or response ◦ Responses: needing a very short time period between the occurrence of the event and the recording of the event Time is an important dimension, because asking users to recall after-the-fact how much time they spent or wasted will lead to inaccurate answers

8. Why use diaries in HCI? Useful if data cannot be accurately collected as we do in experimental research (measuring human performance), or observational means (e.g., automatic data collection) Survey/diary: ask users about themselves ◦ How did they perceive a certain experience with the computer or device? ◦ How did they feel? ◦ How did they respond? ◦ How much time did it take them? ◦ How did it impact on their mood? ◦ When did they use it?

9. Why use diaries in HCI? Diaries can investigate the use of technology that exists in multiple stages ◦ Tech that does not exist but could (where researchers investigate communication or info usage patterns, separate from the tech) ◦ Tech that exists but needs to be improved (how people use existing technology) ◦ Prototypes of new tech that need to be evaluated

10. Why use diaries in HCI? Diaries are good for recording user- defined incidents ◦ When users intended to perform a task, but decided not to do so ◦ When users feel that they have learned something new Diaries are also good at researching situations where users move around and don’t stay in one place ◦ Mobile phones, GPS devices, hand-held tech

11. Challenges with diaries Users sometimes are not introspective and are not even aware of the specifics of what they are doing and therefore may have trouble recording it in a diary entry Users may not follow through and record (via paper or electronic) a sufficient number of entries Time recording may still be less accurate for time diaries than for controlled laboratory setting or automated data collection

12. Challenges with diaries Generally harder to recruit users for a diary study than for something less intrusive, like a survey Since data are both qualitative and quantitative, data analysis may take a long time Hard to strike a balance between a frequent-enough series of diary entries, and infringement on daily activities (user participation may then trail off)

13. Participants for a diary study Like any other type of research, prior planning and testing are a must to ensure a valid outcome First step is to determine who will take part in the study (e.g., demographic, education, computer experience) Try to get a representative group of participants, but it’s more important to have users who can provide useful insight Potential diarists must understand the purpose, be motivated and have competence in using any required technology for diary entries

14. Participants for a diary study Diary study: ◦ Must be structured so that it yields useful data, without imposing an unreasonable burden on users  e.g., employment, health, relationships with others ◦ Should not negatively impact on employment, health, or relationships Participants: ◦ Should be paid for taking part in the diary study ◦ Need to be informed of their rights, including to remain anonymous

15. Types of Diaries Feedback diary: the data recorded in the diary is itself the purpose of the research (i.e., meaningful to the researcher) ◦ Users make entries when a certain event or threshold occurs, or on a stated time basis Elicitation diary: the users record only basic information about important events occurring in their day (i.e., meaningful to the user; also the researcher) ◦ These data points are used as prompts for expansion at a later time

16. Types of Diaries: Feedback diary How often a diary entry is made? ◦ What event, time, or threshold triggers? Examples:  When you feel frustrated with an interface  When you completed a certain task Users can also make a recording at a set time every day (say 9PM), or during a specific time period (say 12-6PM) Feedback diaries can be structured or unstructured ◦ Likert scales, checkboxes, time recording (e.g., every 15 minutes) ◦ General reflection (“How are you feeling right now about your computer?; How did you respond? How did you think the tech could be improved?”)

17. Types of Diaries: Elicitation diary Data points recorded in an elicitation diary are quick and simple ◦ Short snippets of text ◦ Short audio records ◦ Short video clips or pictures Should not interrupt what users were doing Then, later (in an interview, on a web site, or other format), users expand on their entries, describing in more detail

18. Elicitation diary (examples) Development of a new handheld document scanner (Brown et al. 2000) ◦ 22 users were asked to take a photo any time they felt that there was a document they would like to capture ◦ Over 7 days, they made 381 entries ◦ Pictures were then used during a series of semi- structured interviews to prompt users Mobile information seeking (Sohn et al., 2008) ◦ Users were asked to send in a short text message, identifying when they had an info need ◦ This short text entry is then used to remind the users of answering a series of questions on a web site (e.g., Where were you? What were you doing?)

19. Types of Diaries: Hybrid approach Example: student’s use of transportation (Carter and Mankoff, 2005) ◦ For two weeks, users were asked to call a specific number every time they made a transit decision (location is automatically captured using GPS) ◦ During a later interview, the users were presented with the recordings and were prompted to expand their thoughts on specific decisions

20. Cf) Experience Sampling Capture participants’ in-situ responses to hypothetic requests ◦ Precisely controlled timing ◦ Tracking compliance ◦ Characteristics of rating behavior (w/ additional data collection) ◦ Reduction of human error (in data management) Example: understanding location disclosure (Consolvo et al., CHI 2005) ◦ Phase1: identify a buddy list ◦ Phase 2: participants receive 10 randomly timed questionnaires everyday for two weeks from 9AM-9PM on weekdays; 10AM- 10PM on weekends  Questionnaire: Context (Where are you? What are you doing? With whom are you?) + What the user wants to disclose about location (based on the hypothetical request from a social relation on the buddy list) ◦ Phase 3: participants reflected their experiences in Phase 1&2

21. Data collection How will the diaries be recorded? ◦ Paper? ◦ Electronic? ◦ Voice? ◦ Smart phones? Often now, technology is being used to record diary entries Crucial to choose the media that will be most natural for the diarists in their everyday life Two weeks is often an appropriate length of time for a diary study

22. When to record an entry? Participants should be given clear guidance on when to perform an entry in the diary ◦ What activities are of interest? ◦ What events, incidents, or feelings should result in a diary entry? ◦ How often should diary entries be made? ◦ How detailed should the entries be? Make sure NOT to pay participants based on the number of diary entries ◦ Any payment should be for regular participation but should not be linked directly to # entries

23. When to record an entry? If diary reports are turned in during the study period, monitor the incoming reports (e.g., who’s not reporting or not providing useful data) Reminders and feedback can be sent during the period of the diary study ◦ e.g., 5 text messages were sent a day to remind users to send in diary entries (Sohn et al. 2005)

24. Analysis of diaries Transfer all records to an easy-to-analyze electronic format Do statistical analysis on quantitative data Prepare and examine qualitative data, potentially do a content analysis With qualitative data, you can contact the participants after the fact to see if your interpretations are correct

25. Summary Why use diaries in HCI? Challenges with diaries Participants for a diary study Types of Diaries (feedback, elicitation) Data collection When to record an entry? Analysis of diaries