1. Human-Computer Interaction A Computer Science Perspective Benjamin Lok September 20 th, 2004

2. Outline HCI HCI Computer Science take Computer Science take Research Research My Research My Research Virtual Reality Virtual Reality Interactive Virtual Characters Interactive Virtual Characters

3. HCI Community Academics/Industry Research Academics/Industry Research Taxonomies Taxonomies Theories Theories Predictive models Predictive models Experimenters Experimenters Empirical data Empirical data Product design Product design Other areas (Sociologists, anthropologists, managers) Other areas (Sociologists, anthropologists, managers) Motor Motor Perceptual Perceptual Cognitive Cognitive Social, economic, ethics Social, economic, ethics

4. Computer Science Preliminary Preliminary Evaluate interfaces Evaluate interfaces Design new approaches Design new approaches Command Line->Direct Manipulation Command Line->Direct Manipulation Current Current New interfaces New interfaces Mobility, Immersion, Mobility, Immersion, Helping companies develop better products Helping companies develop better products Hardware, software, interaction technology Hardware, software, interaction technology

5. Computer Science HCI How do you design interfaces to systems for: How do you design interfaces to systems for: Users with disabilities Users with disabilities Children Children Elderly Elderly Culture and international diversity Culture and international diversity Cognitive diversity Cognitive diversity Physical Physical Universal usability Universal usability

6. Case Study: Library of Congress Database Design http://catalog.loc.gov/ http://catalog.loc.gov/ http://catalog.loc.gov/ Two interfaces Two interfaces Catalog New Books Catalog New Books 3-6 hour training course - staffers 3-6 hour training course - staffers Search Catalog of Books Search Catalog of Books General public – too complex, command language and complex cataloging rules General public – too complex, command language and complex cataloging rules Solution Solution Touch screen Touch screen Reduced functionality Reduced functionality Better information presentation Better information presentation Eventually Web based interface Eventually Web based interface Same database and services, different interfaces Same database and services, different interfaces

7. Example Five fastest places to click on for a right-handed user? Five fastest places to click on for a right-handed user?

8. Example What affects time? What affects time?

9. Human Computer Interaction We see this all the time. We see this all the time. What’s good about the design of this error box? What’s good about the design of this error box? The user knows there is an error The user knows there is an error What’s poor about the design of this error box? What’s poor about the design of this error box? Discouraging Discouraging Not enough information Not enough information No way to resolve the problem (instructions or contact info) No way to resolve the problem (instructions or contact info)

10. My Choice iPod by Apple Computers iPod by Apple Computers Pros: Pros: portable portable power power ease of use ease of use # of controls # of controls Cons: Cons: scratches easily scratches easily no speech for car use no speech for car use proprietary proprietary

11. HCI Tools Sound Sound 3D 3D Animation Animation Video Video Devices Devices Size (small->very large) Size (small->very large) Portable (PDA, phone) Portable (PDA, phone) Plasticity Plasticity Context sensitive/aware Context sensitive/aware Personalizable Personalizable Ubiquitous Ubiquitous

12. Usability Requirements Goals: Goals: Usability Usability Universality Universality Usefulness Usefulness Achieved by: Achieved by: Planning Planning Sensitivity to user needs Sensitivity to user needs Devotion to requirements analysis Devotion to requirements analysis Testing Testing

13. Bad Interfaces Encumbering Encumbering Confusing Confusing Slow Slow Trust (ex. windows crashing) Trust (ex. windows crashing) What makes it hard? What makes it hard? Varies by culture Varies by culture Multiple platforms Multiple platforms Variety of users Variety of users Think of a game you’ve played with a bad interface Think of a game you’ve played with a bad interface UNIX UNIX

14. Standardization, Integration, Consistency, Portability Standardization – common user-interface features across multiple applications Standardization – common user-interface features across multiple applications Apple Apple Web Web Windows Windows Integration – across application packages Integration – across application packages file formats file formats Consistency – common action sequences, terms, units, layouts, color, typography within an application Consistency – common action sequences, terms, units, layouts, color, typography within an application Portability – convert data and interfaces across multiple hardware and software environments Portability – convert data and interfaces across multiple hardware and software environments Word/HTML/PDF/ASCII Word/HTML/PDF/ASCII

15. Accommodating Hardware and Software Diversity Support a wide range of hardware and software platforms Support a wide range of hardware and software platforms Software and hardware evolution Software and hardware evolution OS, application, browsers, capabilities OS, application, browsers, capabilities backward compatibility is a good goal backward compatibility is a good goal Three major technical challenges are: Three major technical challenges are: Producing satisfying and effective Internet interaction (broadband vs. dial-up & wireless) Producing satisfying and effective Internet interaction (broadband vs. dial-up & wireless) Enabling web services from large to small (size and resolution) Enabling web services from large to small (size and resolution) Support easy maintenance of or automatic conversion to multiple languages Support easy maintenance of or automatic conversion to multiple languages

16. HCI Goals Influence academic and industrial researchers Influence academic and industrial researchers Understand a problem and related theory Understand a problem and related theory Hypothesis and testing Hypothesis and testing Study design (we’ll do this!) Study design (we’ll do this!) Interpret results Interpret results Provide tools, techniques and knowledge for commercial developers Provide tools, techniques and knowledge for commercial developers competitive advantage (think ipod) competitive advantage (think ipod) Raising the computer consciousness of the general public Raising the computer consciousness of the general public Reduce computer anxiety (error messages) Reduce computer anxiety (error messages) Common fears: Common fears: I’ll break it I’ll break it I’ll make a mistake I’ll make a mistake The computer is smarter than me The computer is smarter than me HCI contributes to this! HCI contributes to this!

17. Near & Future Interfaces Let’s review Let’s review Minority Report Minority Report Steel Battalion Steel Battalion Eye Toy Eye Toy Dance Dance Revolution Dance Dance Revolution Time to learn Speed of performance Rate of errors Retention over time Subjective satisfaction

18. Overview Computer generated characters and environments Computer generated characters and environments Amazing visuals and audio Amazing visuals and audio Interacting is limited! Interacting is limited! Reduce applicability? Reduce applicability? Goals: Goals: Create new methods to interact Create new methods to interact Evaluate the effectiveness of these interaction methods Evaluate the effectiveness of these interaction methods Aki from Final Fantasy: The Spirits Within Walking Experiment PIT - UNC

19. Collaborators Xi Yong Wang, Aaron Kotranza, Benjamin Lok University of Florida Danette Allen NASA Langley Research Center

20. Virtual Environments Been around for almost 30 years Been around for almost 30 years # of systems in research labs > day to day use # of systems in research labs > day to day use Why? Why? Interaction with the virtual environment is too poor Interaction with the virtual environment is too poor Everything is virtual isn’t necessarily good Everything is virtual isn’t necessarily good Example, change a light bulb Example, change a light bulb Approach: Approach: Real objects as interfaces to the virtual world Real objects as interfaces to the virtual world Merge the real and virtual spaces Merge the real and virtual spaces Evaluate what VR is good for! Evaluate what VR is good for!

21. Getting real objects into VR to aid engineering design Getting real objects into VR to aid engineering design Collaboration w/ Mars Airplane (Langley Research Center) Collaboration w/ Mars Airplane (Langley Research Center) Get tools, parts, and other (possibly distributed) collaborators in a shared space Get tools, parts, and other (possibly distributed) collaborators in a shared space

22. Immersive Virtual Characters for Educating Medical Communication Skills J. Hernendez, A. Stevens, D. S. Lind Department of Surgery (College of Medicine) Department of Surgery (College of Medicine) M. Duerson Department of Community Health and Family Medicine (College of Medicine) Department of Community Health and Family Medicine (College of Medicine) K. Johnsen, R. Dickerson, A. Raij, B. Lok Department of Computer and Information Science and Engineering (College of Engineering) The University of Florida, Gainesville, FL The University of Florida, Gainesville, FL J. Jackson, M. Shin Department of Computer Science The University of North Carolina at Charlotte, Charlotte, NC

23. What is a Virtual Character? Virtual character - a character who represents the state of a system Virtual character - a character who represents the state of a system In TRON (1982), humans and humans that represents software interacted within a world that represented the hardware. In TRON (1982), humans and humans that represents software interacted within a world that represented the hardware.

24. What is a Virtual Character We look to to have humans and human the represent software interact in the real world. We look to to have humans and human the represent software interact in the real world. http://movies.yahoo.com/shop?d=hv&id=1807432839&cf=trailer

25. Motivation “Doctor, I have a pain in my side!” “Doctor, I have a pain in my side!” What you do next depends on: What you do next depends on: Age Age Gender Gender Ethnicity Ethnicity Visual Cues Visual Cues Audio Cues Audio Cues

26. Motivation Medical Diagnosis Components Medical Diagnosis Components Patient Interview Patient Interview Physical Examination Physical Examination Current Teaching Methodologies Current Teaching Methodologies Books Books Standardized Patients (actors) Standardized Patients (actors) Results in Results in Minimal training frequency Minimal training frequency Minimal scenario variety Minimal scenario variety Lack of immediate feedback Lack of immediate feedback Medical students are not adequately prepared for many diagnosis scenarios Medical students are not adequately prepared for many diagnosis scenarios

27. Solution Interactive Virtual Characters Interactive Virtual Characters Life-Sized Life-Sized Computer Generated Computer Generated Natural Interaction Natural Interaction Responds to User Responds to User Responses based on accepted medical procedure Responses based on accepted medical procedure COTS Equipment: COTS Equipment: Projectors Projectors Laptops Laptops Web cameras Web cameras Tablet PC Tablet PC

28. Solution Interactive Virtual Characters Interactive Virtual Characters Computer Generated Computer Generated Natural Interaction Natural Interaction Responds to User Responds to User Responses based on accepted medical procedure Responses based on accepted medical procedure Results in: Results in: More Scenarios More Scenarios More Training More Training Standardized Experiences Standardized Experiences

29. Why do we want digital characters Propose: Digital Characters as a new (meta-) medium to interact with information Propose: Digital Characters as a new (meta-) medium to interact with information Why would we want a digital character? Why would we want a digital character? Effective Interaction Effective Interaction Better than keyboard and mouse for certain tasks Better than keyboard and mouse for certain tasks Dynamic (output easily augmentable) Dynamic (output easily augmentable) 3D 3D Natural interaction Natural interaction Low Bandwidth Low Bandwidth Effective Collaboration Effective Collaboration Controlled conveyed visual information Controlled conveyed visual information Non-verbal communication (60%) Non-verbal communication (60%)

30. Let’s look at interaction Each participant in a communication has three stages: perception, cognition, and response Each participant in a communication has three stages: perception, cognition, and response Define interaction as both the input and output Define interaction as both the input and output Thinking Participant Thinking Digital Character Thinking PerceivingResponding Digital Character Responding PerceivingResponding Perceiving Interaction

31. Combines Speech Speech Computer Vision Computer Vision Eye Gaze Eye Gaze Gesture Recognition Gesture Recognition Repeat your gestures Repeat your gestures High Quality 3D models High Quality 3D models Animation Animation Rendering Rendering Visualization Visualization

32. Integrates Computer Science Computer Science Computer Graphics Computer Graphics Image Processing Image Processing Natural Language Processing Natural Language Processing Medicine Medicine Training Training Education Education Standardization Standardization Education Education Multimedia Learning Multimedia Learning Technology based Learning Technology based Learning Training Training What other areas could this be used for? What other areas could this be used for? Psychology Psychology Social Science Social Science Education Education

33. Current and Future Work Current Status: Current Status: Initial scenario created with gesture, speech, and visualization components integrated Initial scenario created with gesture, speech, and visualization components integrated Evaluating with a group of Medical Students Year 2 Evaluating with a group of Medical Students Year 2 Future Work: Future Work: Formal evaluation studies Formal evaluation studies Increase and improve scenarios Increase and improve scenarios Enhance interactivity Enhance interactivity Show video Show video