ISTD 2003, Audio / Speech Interactive Systems Technical Design Seminar work: Audio / Speech Ville-Mikko Rautio Timo Salminen Vesa Hyvönen

ISTD 2003, Audio / Speech Introduction When gathering information about surrounding environment, hearing is one basic sense for humans. Therefore, usage of audio and speech as an alternative input and output method can effort a lot to a user experience in interactive systems and make it more natural.

ISTD 2003, Audio / Speech Motivation Building interactive systems, user interface should behave according to the expectations of the user experiences of the real world. Generally, user interfaces today are mainly based on keyboard and screen. Feedback from system is given basically only in visual form. In computer-based systems, much better user experience can be achieved by offering information using also other basic senses such as hearing, sense of taste, touch and smell.

ISTD 2003, Audio / Speech Implementation Basically two components: Audio playback and speech/audio recognition. Design issues: Audio can be speech / non-speech To whom are you designing for? Different users – different abilities Blind, old and disabled people Human diversity – physical, perceptual, cultural and intellectual differences Mobile computing Limited input, limited output, slow processor, small memory, limited battery life, slow network connection Communication protocol Speech recognition causes major problems Accuracy Usage in critical systems?

ISTD 2003, Audio / Speech Applications MIT Media Lab – Nomadic Radio: Wearable Audio Computing A client-server based messaging infrastructure utilizes spatialized audio, speech synthesis and recognition hourly news broadcasts, voice mail, , calendar reminders, weather forecasts, stock reports are delivered HP Labs – SpeechBot a search engine for audio & video content that is hosted and played from other websites using speech recognition

ISTD 2003, Audio / Speech Nomadic Radio Network Architecture

ISTD 2003, Audio / Speech

Strengths / Advantages Data input possible without keyboard. Mobile devices Excellent for hands/eyes busy – situation.

ISTD 2003, Audio / Speech Strengths / Advantages People with visual or other disabilities Natural way for humans to interface with the environment Increase the bandwidth of communication Devices with limited screen – need for additional output method Technology available now

ISTD 2003, Audio / Speech Limitations / Weaknesses Input is error prone especially in noisy environments Vocabulary size in recognition - Controlling objects and things is limited Communication protocol needed “Computer! Shut down the lights!” Can lead to unnatural experience How to tell user what communication protocol is like: Explicit – tell exactly what to say (“Welcome to library, say “XXX” to...”) Implicit – open ended, potential for errors (“Welcome to library, what would you like to do….”).

ISTD 2003, Audio / Speech Limitations / Weaknesses Speech output sounds unnatural Asymmetrical speech input is faster than typing speech output is slower than reading Feedback & latency User needs to know if recognition was successful Is system processing data or waiting input? Time taken to recognise utterance Pauses

ISTD 2003, Audio / Speech Selected Industrial Players IBM Conversational Biometrics Combines multiple verification sources such as voice biometrics with spoken knowledge. Embedded ViaVoice IBM speech technology to mobile devices Command and control (C&C) Text-to-Speech (TTS) Sony SDR-4X Prototype of entertainment robot using multi-modal human interaction technology Individual person detection by the tone of voice Continuos speech recognition and unknown vocabulary acquisition Speech synthesis and singing voice production

ISTD 2003, Audio / Speech SDR-4X

ISTD 2003, Audio / Speech Selected International Research Groups and Projects The MBROLA Project Develops speech engine which synthesizes written text for many different languages Speech Engine core freely available! Stanford University – Interactive Workspaces Goal is to create interactive space where you can work collaboratively using natural gestures Speech Interface Group, MIT Media Laboratory Major player, numerous projects Example: Nomadic Radio: Wearable Audio Computing

ISTD 2003, Audio / Speech Selected International Research Groups and Projects MIT, PROJECT OXYGEN Pervasive, human-centered computing Integrated software system that will reside in the public domain Speech and vision, provide the main modes of interaction in Oxygen. Multilingual systems support dialog among participants speaking different languages. The SpeechBuilder utility supports development of spoken interfaces.

ISTD 2003, Audio / Speech

Selected Finnish Research Groups and Projects VTT, Interactive Intelligent Electronics (IIE) User interface technologies for future home environments, The Smart-Its Project, Beyond the GUI, … Helsinki University of Technology, Neural Network Research Centre Adaptive Natural Language Processing Tampere University of Technology, Speech-based and Pervasive Interaction Group USIX-Interact, Dumas, Mobile User Interfaces, …

ISTD 2003, Audio / Speech Companies and Research Groups in Oulu MediaTeam Oulu, Language and Audio Technology CBIR – Content Based Information Retrieval Filling of the Semantic Gap in the Retrieval of Audio and Video Recordings Multiparametric prosodic analysis of phonetic and phonological correlates of emotions Vikings

ISTD 2003, Audio / Speech Future Developments Multimodality Multilingual, natural speech interaction Emotional state Biometrics