MIT Media Lab Contextual Awareness and Messaging in Nomadic Environments Design for Wearable Audio Computing Nitin Sawhney Ericsson Eurolabs Deutschland GmbH August 24th, 1998 Speech Interface Group MIT Media Laboratory, Cambridge, USA

MIT Media Lab Outline of Presentation Personal Augmentation Landscape (80’s, 90’s, future) Prior work in Speech Group & Research Challenges Wearable Audio Computing & Design of Nomadic Radio Voice Navigation, Ambient Awareness and Spatial Audio Contextual and Scaleable Notification Ongoing Research and Future Work

MIT Media Lab Personal Augmentation Landscape (early 80’s) ActivePassiveDemand on User’s Attention Resources Situational Awareness Low High Pagers Watch Eye-glasses Pen-paper Walkman Desktop PCs Airplane Cockpit Automobiles

MIT Media Lab ActivePassiveDemand on User’s Attention Resources Situational Awareness Low High Pagers Mobile-Phones Watch Eye-glasses Pen-paper Walkman HMD Wearables Network PCs Automobiles Airplane Cockpit Newton Virtual Reality Personal Organizers Personal Augmentation Landscape (early 90’s)

MIT Media Lab Telephone-based Messaging Systems Phoneshell - Chris Schmandt ‘93 Chatter - Eric Ly ’94 NewsPhone - ‘96 MailCall - Matt Marx ‘96 Industry Products: WildFire, AT&T PocketNet, Nokia Communiator, General Magic Porticio

MIT Media Lab Mobile Audio Devices Voice Notes - Lisa Stifelman ’93 NewsComm - Deb Roy ’95 Audio Notebook - Lisa Stifelman ‘96 Dynomite - Sawhney & FXPAL ‘96 Industry: Digital Voice Recorders Internet Audio Players, IBM CrossPad and Telematics

MIT Media Lab Wearable (audio) Computing Ubiquitous Talker - Sony ‘95 Speech Wear - CMU ’96 Audio Aura - Xerox PARC ’97 Remembrance Agent - Thad Starner and Brad Rhodes ’96 (MIT Media Lab) DIE AUDIO GRUPPE - electro-acoustic clothes ‘83-98

MIT Media Lab ActivePassiveDemand on User’s Attention Resources Situational Awareness Low High Pagers Smart-Phones Watch Palm-Pilot Eye-glasses Pen-paper Nomadic Radio Walkman HMD Wearables COMRIS Audio-Aura Desktop PCs Automobiles Airplane Cockpit Telematics Newton Mobile Audio Virtual Reality Speech Wear Personal Augmentation Landscape (1997-future)

MIT Media Lab Research Challenges for Nomadic Augmentation Unified Access to Heterogeneous Information Unobtrusive & Peripheral Interfaces Knowing when and How to Interrupt the User Sensing Environmental Context Adaptive Learning of Usage Patterns

MIT Media Lab Nomadic Radio: Design Overview Nomadic Radio Integrated Audio Interface for Personal Messaging on a Wearable Platform Audio Cues for Notification Voice Recognition for Navigation and Control Spatial Audio for Simultaneous Listening "New short personal message from Walter about Where do we go for Lunch?" Synthetic Speech for Feedback Ambient Sound for Awareness

MIT Media Lab Unifying Information Services via Dynamic Views and Modeless Interaction

MIT Media Lab System Architecture

MIT Media Lab Wired Wearable vs. Distributed Wireless Mode Nortel Soundbeam Neckset VIA FlexiPC MIT Media Lab Lizzy Toshiba Libretto Wireless Speech Input Networked Speech Rec. PC Wireless 3D Audio Spatial Audio Speech Synthesis PC Wavelan Tactile Input

MIT Media Lab Radio Vest: Alternative Wearable Design Conceptual design and final implementation of an outdoor wearable configuration with clip-on directional speakers and powered microphone Radio Vest Alternative Wearable Audio Design

MIT Media Lab Listening and Speaking on the Run: A Scenario

MIT Media Lab Speech I/O and Voice Navigation "Last personal message from Geek about help" "Important message 7 from Tony about Gym?" "Group message 6 from Hiroshi about CHI'98" Temporal Target Window for message 7 1 sec. 2 sec. extension Message 8 Scanning Activated Message 6Message 7 Time Temporal Target Windows Always on, Listening and Easily Woken-up Networked Implementation of AT&T Watson Speech API Push-to-Talk vs. Continuous Monitoring Navigation: Browsing vs. Scanning

MIT Media Lab Awareness & Notification via Auditory Cues PriorityFeedbackAwarenessIdentification

MIT Media Lab Simultaneous and Spatial Listening distance azimuth elevation spatial audio source 12:00 PM 6:00 PM 9:00 PM3:00 PM 2:30 PM horizontal plane Localization using an HRTF ModelChronological 12-hour Spatial Layout Spatial Memory Stream Segregation Mapping Time to Space

MIT Media Lab Message Swatch: Mapping Time to Space Voice Message Upcoming Calendar Event TheMessage Swatch, displaying voice messages and as concentric beams and calendar events as arcs. Message Swatch Spatial Scanning

MIT Media Lab Notifications in Everyday Environments Binary Notification Cues No Awareness of User and Environment No Adaptive Learning from Prior Interactions No Unified Notification strategy between devices Solution: Scaleable & Contextual Notification Pager PDA Cell-phone

MIT Media Lab Scaleable Presentation and Dynamic Operational Modes "Hi mom, its Kathy." "new voice msg from Kathy." "Hi mom, its Kathy. Can you pick me up early from school today?" SilenceAmbientAudio CuesPreviewFull BodyForegroundSummary "Hi mom, its Kathy. Can you pick me up early from school today?"

MIT Media Lab Contextual Notification Model

MIT Media Lab Environmental Audio Classification Multi-user Awareness and Communication Protocols Attention-Cognition Research & Evaluation Affective Attention via EEG Sensing & Analysis Vision: Ubiquitous Autonomous Entities (Devices and Agents) with Cooperative and Evolving Context-Attention Models Ongoing Research and Future Vision

MIT Media Lab Contextual Awareness and Messaging in Nomadic Environments Design for Wearable Audio Computing Nitin Sawhney Speech Interface Group