1. CEN3722 Human Computer Interaction Advanced Interfaces
Dr. Ron Eaglin

2. Objectives Define multi-modal interfaces
Disadvantages and advantages of using speech in interfaces
Define speech synthesis
Define and provide examples of non-speech sound
Describe problems of using hand-writing recognition and computer vision as modes in systems
Describe virtual reality and how it might be used to enhance systems
Describe advances in computing and HCI needs

3. Multi-modal interfaces
We have many senses, but use only visual (little auditory) in most computer systems
Develop multi-modal and multimedia systems to take advantage of our other senses

4. Speech Speech is a natural form of human communication
Difficult for computers
Language is:
Complex
Context-dependent
Varied

5. Speech Recognition Barriers to success Background noise
Natural continuation
“umm…”, “errr…”
Variations between individuals
Different inflections
Different euphemisms
Different regional accents

6. Speech Advantages Disabled Jobs requiring hands
Visual
Physical disabilities
Dyslexia
Jobs requiring hands
Factory workers
Jobs requiring visual focus (speech input still OK)

7. Speech Synthesis Can use speech as output
Provide for more natural, human-like conversation capabilities
Problem:
Understanding and meaning can be highly dependent upon variations in intonation.
System must understand the problem domain
Canned messages

8. Uninterrupted Speech
Fixed prerecorded messages to supplement or replace visual information
Because recorded, have natural prosody and pronunciation
Announcements at airports
Automatic pilot

9. Non-speech Sound
Traditionally used to provide warnings, alarms and status information.
Dual-mode displays thought to be better
Presentation of information along different, parallel channels allows brain to search along two paths
Presentation of redundant information may increase user’s performance
May remember sound, not visual, or vice versa.

10. Non-speech Sound Traditionally UI almost entirely visual
Potential to overload visual channel
Force user to attend to too many things at once
Persistence of visual information means that even detail that is quickly out of date may remain on display after it is required

11. Non-Speech Sound
Speech is serial, we must listen to most of sentence to extract meaning
Make take a long time
Non-speech sound
Can be associated with a particular action and assimilated in a much shorter period of time.
Can also be universal, like icons; not true of speech.
Must be learned, whereas speech doesn’t (as long at the listener knows the language!)

12. Non-speech Sound Non-speech sound
Is short-lived, so can provide transitory information
Provide status on background processes
Provide a second representation of actions and objects to support the visual mode and provide confirmation to the user.
Navigation

13. Handwriting Recognition
Writing as an input medium
A natural form of communication
Digitizing tablet used to capture input
Problems
Similar to speech, variations in individuals
Writing varies from day to day for individuals

14. Computer Vision Additional input channel for the computer
Video camera and software used to identify user and tailor system to perceived requirements
Bottom-up approach
Pixels progressively analyzed to extract meaning
More complex problem than speech or handwriting recognition

15. Ubiquitous Computing
Computers on the desktop to computers “everywhere”
Intention: “… to create a computing infrastructure that permeates our physical environment so much that we do not notice it anymore”
Invisible Computing
Analogy with motors
Size depends on use
Are invisible

16. Ubiquitous Computing Automated capture, integration, access
Meeting room environments
Note taking
Context-aware computing
Most “ubiquitous computing” miniaturized desktop computing
Location awareness
Other
Scalable interfaces
Unobtrusive ubiquity

17. Users with Special Needs
Input/output capabilities driven by the “average user”
Leaves out entire groups of people
Physically challenged
Sight
Hearing
Limbs
GUI and the sight-challenged

18. Virtual Reality Computer generated simulation of a world
Specialized hardware and software (headset)
Use of multi-modal devices
Heavily used in training systems (military)
Expanding rapidly into entertainment industry
Lagging in many industries – still an emerging market (2016)

19. Artificial Intelligence (AI)
Advances in AI open up an new entirely new field of HCI
Interfaces for self-driving vehicles (application of AI)
Customer service AI system in use today
Systems like IBM Watson can process incredible amounts of information – provide (with correct interface)
Can write applications for Watson (ibm.com/Watson)

20. Personal Robotics
Interaction with personal robotics now being defined