1. Aiming Computing Technology at Enhancing the Quality of Life of People with ALS Some Sketches on Directions in Minimal Signaling Communication Communication Working Group Microsoft Research & ALS Society of British Columbia

2. Directions Tools and method for planning and creating long- term corpora and strategies for communication Creating effective input strategies and systems for different phases of the progression of ALS Exploration of novel signaling strategies, such as eye gaze, as an effective input modality Coupling contextual information as source of evidence in reasoning about intentions Potential for integrated robotic systems for motion Multiple directions

3. Core Challeges & Opportunities with Gaze-Centric Systems Move beyond potentially frustrating point and dwell approaches Toward new UI designs, metaphors & control methodologies that can ease user effort and enhance accuracy of selection, control. Adaptive techniques, use of context can ease user effort and error. Promise of creating multiple rewarding and valuable applications, for communication, creative expression, information access, enjoyable experiences, control of real- world sensors and effectors.

4. Sample Applications

5. Example Gaze-centric interaction with (previously authored) databases of images and audiovisual snippets for enhanced communication

6. Memories library from video and image photolibrary Social discourse library of videos and stills for sharing emotions, moods, gestures.

7. Example Gaze-centric control of confirmation, negation, selection of alphabetic controls for writing, editing.

8. Yes No Big Time! No way. [ask] [tell] [have] Q W E R T Y U I O P A S D F G H J K L Z X C V B N M. ? [ I want to … Predictive language model with correction

9. Example Gaze-centric control of integrated, supportive robotic systems – Head motion – Head gestures (nods, yes; shakes no) – Arms (and hand?) motion …etc.

10. Turn head left or right (e.g., in increments)

11. Conversation Confirmation Nod “yes”; shake head “no” (Beyond confirmation: nod as a natural cue for understanding in stream with conversation)

12. Toward more general application of “integrated robotics,” e.g., arms controls, etc., for atomic and patterned moves (e.g., wave hello).

13. Example Gaze-centric control of navigation through cached real-world, or 3D virtual-reality animation sequences, potentially with other appropriate sensory stimulation.

20. Example Gaze-centric communication and linking for single and multiplayer games, including chess, card playing, virtual worlds (Second Life, etc.)

21. Example Gaze-centric control of telepresence applications for real-time viewing from different viewpoints.

22. Example Gaze-centric control of painting, CAD, sculpting machinery, machine lathes, other artistry and crafts.

23. Longer-Term: Addressing Totally Locked in State (TLS) Challenge of use of EEG for preference assessment, acknowledgment, confirmation, selection E.g., Rotating selection rings, with “other” option; navigation among hierarchy of rings, use of simple input for halting, selecting, confirmation.

24. Multiple Directions and Possibilities Comments? horvitz@microsoft.com http://research.microsoft.com/~horvitz