1. Human Computer Interaction
Eugene Annabel
Mary Curtin
James Lawler
Marilyn Mead
Pauline Mosley
Overview
11/27/2018

2. Agenda HCI Overview 3-D Audio for HCI Perceptual User Interfaces
Perceptual Intelligence
Helping People Find What They Don’t Know
Wearable Devices
Overview
11/27/2018

3. HCI Overview Perceptual User Interfaces Big Picture
Perceptual User Interfaces Individual Topics
Text
Text
Text
Text
Text
Text
Text
Text
Text
Overview
11/27/2018

4. Three revolutions in computing
1950 – mainframe computing:
one computer used by many people
1975 – PC computing:
one computer used by one person
2000 – ubiquitous computing:
many computers used by one person
(goal to be nearly invisible)
Overview
11/27/2018

5. Good technology is invisible
“invisible” technology stays out of the way of the task
bad technology draws attention to itself, not the task
ubiquitous computing is about “invisible” computers
Overview
11/27/2018

6. What is the interface agent metaphor?
You talk to it
It watches us and learns our needs
It has “knowledge”, is “aware”, or has a personality
It is an assistant
Overview
11/27/2018

7. Limitations of the interface agent
It does not go far enough
It keeps the computer in the foreground
It stays within an old paradigm
It obsessively fascinates
Overview
11/27/2018

8. Vocabulary
Pervasive Computing – refers to a world where people can access information, take action and conduct transactions from any place with any device. I think of it as any device, any network and any content
Haptics – refers to sensing and manipulation through the sense of touch
Overview
11/27/2018

9. Acronyms HCI – human-computer interfaces
GUI – graphical user interfaces
WIMP – windows, icons, menus, pointers
PUI – perceptual user interfaces
Overview
11/27/2018

10. 3-D Audio for HCI
Richard O. Duda
Overview
11/27/2018

11. 3-D Audio for HCI The physics of sound
The psychoacoustics of spatial hearing
Simple spatial audio systems
Head-related transfer functions
HRTF – based spatial audio systems
References
Overview
11/27/2018

12. The Physics of Sound
Basic Concepts (propagation, reflection, refraction)
Sine Waves (amplitude, frequency and wavelength)
Fourier Analysis
Linear Systems (impulse response and transfer functions)
Logarithmic Scales (db and octaves)
Overview
11/27/2018

13. Psychoacoustics of Spatial Hearing
Coordinate Systems
Azimuth Cues
Elevation Cues
Range Cues
Reverberation and Echoes
Overview
11/27/2018

14. Simple Spatial Audio Systems
Two-channel (stereo)
Multichannel
Binaural recordings
Overview
11/27/2018

15. Head-Related Transfer Functions (HRTF)
Head Related Impulse Response (HRIR)
Apply Fourier Transform to get Head Related Transfer Function
Purpose – to synthesize accurate binaural signals from a monaural source
Overview
11/27/2018

16. HRTF Based Systems The Convolovotron – a spatial audio system
Headphones versus loudspeakers
The need for head tracking
Measured versus modeled HRTF’s
Models for HRTF’s
Overview
11/27/2018

17. Perceptual User Interfaces
Matthew Turk, George Robertson
Overview
11/27/2018

18. Perceptual User Interfaces
Human-computer interaction has not changed fundamentally for nearly 2 decades.
Typing
Pointing
Clicking
Overview
11/27/2018

19. Human Computer Interfaces
GUIs
User is provided commands and action with known effects
User has sense of responsibility about interactions with computer applications
Overview
11/27/2018

20. WIMP – stable and global
Windows
Icons
Menus
Pointer
Overview
11/27/2018

21. Computer Devices Smaller Ubiquitous Pervasive
Convergence between computers and TV
Pointing, clicking, typing not adequate
Overview
11/27/2018

22. Perceptual User Interfaces
Interaction techniques
Combine understanding of natural human capabilities with computer I/O with
Machine perception and reasoning
Transparent and passive devices and sensors
Machines perceive relevant human communications to generate naturally understood output
Overview
11/27/2018

23. Integration at multiple levels of technologies
Speech and sound recognition and generation
Computer vision
Graphical animation and visualization
Language understanding
Overview
11/27/2018

24. Integration at multiple levels of technologies
Touch-based sensing and feedback
Learning
User modeling
Dialogue management
Overview
11/27/2018

25. Perceptive UI Adds human-like perceptual capabilities to computer
Computer is aware
What user is saying
What user’s face, body, hands are doing
Overview
11/27/2018

26. Multimodal UI Emphasis on human communication skills
Using speech with pen-based gestures
Modalities: visual display, audio, tactile feedback
Focuses on human perceptual channels
Overview
11/27/2018

27. Multimedia UI
Uses perceptual and cognitive skills to interpret information presented to user
Text, graphics, audio, video typical media
Focuses on the media
Overview
11/27/2018

28. PUI Integrates perceptive, multimodal, multimedia interfaces
Creates more natural and intuitive interfaces
Enhances use of computers as tools
Enhances GUI-based applications
Use computers as assistants or agents to interact in human-like ways
Overview
11/27/2018

29. Challenges of PUI Ambitious with diverse elements
Extensive work on speech and pen-based systems
Lead to more stable and robust systems
Overview
11/27/2018

30. Perceptual Intelligence
Alex Pentland
Overview
11/27/2018

31. Perceptual Intelligence
Inanimate things are coming to life
Desks & doors
TVs & telephones
Cars & trains
Glasses changed to smart active helpmates
Computers are isolated
Probabilistic rules – derived by statistical learning
Overview
11/27/2018

32. Goal of Perceptual Intelligence
Not to create computers with logical powers
Not to have computers ubiquitous and networked
To create systems with reliable perceptual capabilities
To create systems with ability to learn simple responses
Overview
11/27/2018

33. Audiovisual Systems
Learn word meanings from natural audio and visual input
Understand and generate spoken language
Language acquisition
Apply to human-computer interfaces that use spoken input
Overview
11/27/2018

34. Testbeds at MIT Media Laboratory
Smart rooms
Smart clothes
Sensors so computer sees, hears, and interprets user’s actions
Cameras,microphones, electromagnetic field sensors
Smart thing knows what is going on and reacts intelligently
Overview
11/27/2018

35. Smart Rooms Where are the people? – tracking
Who is it? – face recognition
Facial expression
Recognize hand and body gestures
Smart rooms have smart chairs
Overview
11/27/2018

36. Examples of Smart Clothes Applications
GPS in belt to navigate
Accelerometers and tilt sensors distinguish walking from standing from sitting
Medical monitors
Feedback on alertness and stress level
Early warning for high-risk medical problems
Overview
11/27/2018

37. Examples of Smart Clothes Applications
Reminds user of relevant facts during conversations
Face recognition software to remember name of person user is looking at
Possible shots in billiards
Sign language
Overview
11/27/2018

38. Conclusion
Possible to track people’s motion, identify them by voice and facial appearance, recognize actions in real time
Smart rooms and smart clothes can recognize people, understand speech, control information display without mouse or keyboard, communicate by facial and hand gesture
Interact in a personalized adaptive manner
Overview
11/27/2018

39. Helping People Find What They Don’t Know
Nicholas J. Belkin
Overview
11/27/2018

40. Helping People Find What They Don’t Know
How a machine can help a user via suggesting recommendations
Typical information systems require user to specify what to retrieve
People are unfamiliar
With underlying operations of the systems
With vocabularies the systems use
With the nature of databases
Information systems should recommend courses of action
Overview
11/27/2018

41. The Problem
Choosing correct (key) words to represent their information problems
User generates a query
Multistage interactive process of initial query formulation
Overview
11/27/2018

42. The Problem
subsequent iterations of query reformulation based on interaction with the system
relevance feedback (John Rocchio)
Overview
11/27/2018

43. System Controlled User generates a query
User indicates relevance or no relevance of a retrieved object
System reformulates the query
List of objects retrieved as a result of reformulated query
Overview
11/27/2018

44. User Controlled
User examines suggested terms to manually reformulate query given information from system
User controls how query is reformulated
Overview
11/27/2018

45. Research group at Rutgers
Investigating support for query reformulation
With respect to relevance feedback versus term recommendation
Better when user informed of basic algorithms used in query reformulation
Result: user were willing to give up control over suggested terms through explicit relevance feedback in favor or reduced effort in the uncontrolled term suggestion
Overview
11/27/2018

46. Wearable Devices – New Ways to Manage Information
Mark Billinghurst and Thad Starner
Overview
11/27/2018

47. Goals of Wearable Computers
Must be mobile
Augments reality
Provides context sensitivity
Overview
11/27/2018

48. Current Wearables Less obtrusive Still awkward
Noticeably apart from everyday wear
Integrated with wearer’s daily life
Augments human intelligence by being the storage and retrieval tool
Overview
11/27/2018

49. A Wearable Computer Can absorb a rich source of contextual information
Contextual information = conversation, location, gesture, ambient sound
Automatic response
Overview
11/27/2018

50. A Wearable Computer
Preloads sender’s phone number into wireless phone software based on wearer’s Rolodex
Supports collaboration
Overview
11/27/2018

51. Examples of Research Boeing Computer Services Symbol Technologies
Isolating and repairing faults on Boeing 757
Symbol Technologies
Finger mounted ring scanner
Arm mounted computer and finger scanner for UPS
Overview
11/27/2018

52. Where to Get More Information
Overview
11/27/2018