1. Evaluation in Experience-Focused HCI: Exploring the Virtual Intimate Object Future Applications Lab Viktoria Institute Gothenburg, Sweden 13 January 2006 Joseph Jofish Kaye Culturally Embedded Computing Cornell Information Science jofish@cornell.edu

2. Evaluation in Experience-Focused HCI Evaluation in experience-focused HCI is the poor cousin of design. As a field (if we are one!) we have excellent theoretical foundations (Dourish, Wright & McCarthy), a wide repertoire of design techniques (Equator Project, IDEO) and extensive tools for building (CS, EE, PD, ID). But traditional HCI evaluation techniques (GOMS, etc) are inadequate for ready-at-hand technologies used in ways unreplicable in the lab.

3. Virtual Intimate Object (VIO) A small red circle in Windows taskbar, or in a Mac window. When circle is clicked, partners circle glows bright red, then fades over time. http://io.infosci.cornell.edu Kaye, Levitt, Nevins, Golden & Schmidt. Communicating Intimacy One Bit at a Time. Ext. Abs. CHI 2005, ACM Press. Kaye. I just clicked to say I love you. alt.chi, Ext. Abs. CHI 2006, ACM Press.

4. Intimacy in HCI History of elegantly designed but ad-hoc interfaces for communicating intimacy in HCI; hard to evaluate Fields & Thresholds (Dunne & Raby DoP2 '94) Feather, Scent & Shaker (Strong & Gaver CSCW'96) The Bed (Dodge CHI'97) UbiComp'03 Workshop on Intimacy www.intimateornot.org Digital Family Portrait (Mynatt et. al. CHI'01) inTouch (Brave, Ishii, Dahley CSCW'98) inStink, Honey Im Home (Kaye '02, interactions '04) ……

5. How much does Ariel love Eric? Count the hearts! Disney (1997) The Special Edition Little Mermaid Coloring Book. Golden Books Publishing. Answer: 19

6. Two VIO Studies Study I –10 couples in existing long-distance relationships (n=20) 5 couples assigned to VIO (n=10) 5 couples assigned to MinIO (n=10) –Paper logbooks; pre, post-tests, daily questions Study II –80+ subjects recruited through long distance relationship communities on LiveJournal –All subjects took pre-survey; all were asked to take post-survey –Four groups: VIO, VIO + daily survey, daily survey alone, nothing –All surveys online- Currently being analyzed

7. Experience surveys: starting points Gaver et. al: Cultural Probes Geertz: Thick description Dourish, Wright & McCarthy, etc: experience-based HCI Iterative design: Design, Build, Evaluate..

8. Thick Description Winking isnt just something your eye does Its culturally embedded Its only the thick description of the context and culture that lets us understand the role of the wink in sharing a conspiracy, or even parodying another sharing conspiracy. Geertz, C. (1973) Interpretation of Cultures Ch. 1

9. Cultural Probes Gives context around a situation Originally for inspiring the design part of the design build evaluate iterative design cycle Repurposed here for inspiring the evaluation part of the cycle Were changing a key, generally unspoken assumption by proposing that evaluation is creative, arbitrary, and complex. Gaver et. al. Cultural Probes, interactions 6(1) 1999

10. Three topics Questions/tasks about the technology being evaluated. Questions/tasks about the situation. Questions/tasks about instrument being used to gather information

11. And three kinds of questions (for organization, not canonical) Context: Where? When? Take a photo of… How long? How often? Metaphor: What TV show? What song? What colour? What metaphor? Valence: Whats the best? Whats the worse? Whats your favorite? What should we change? What shouldnt we change? Draw the best, the worst.

12. Question Design chart ContextMetaphorValence Technology (VIO) Situation (Long distance relationship) Instrument (Probe / Survey)

13. Getting beyond usability Evaluating in context requires working in context VIO becomes ready-at-hand, not present-at-hand, and so evaluation must be in context. Users must have significant, continuous use, and you must have solved the usability problems otherwise most of what you get is usability issues – i.e. Mobile Probes paper Hulkko et. al. Mobile Probes Proc. NordCHI 04

14. Discussion Unpacking meanings of evaluate –…tell me if people like it –…tell me if a variable that I can track has changed –…make a comparison between my stated (unstated?) goals/values/principles and the built technology –…to tell me what to do next When? (Formative, iterative, final, discrete, continuous)

15. Thanks to Maria Håkansson, Lars Erik Holmquist, Phoebe Sengers, the Culturally Embedded Computing Group, my subjects and my coauthors. Joseph Jofish Kaye Culturally Embedded Computing Cornell Information Science jofish@cornell.edu This talk at jofish.com/talks. VIO at io.infosci.cornell.edu