Attentive User Interfaces by Falk Fleischer

Overview Introduction Principles & Methodologies Applications Summary Attention - Sensitive Messaging Mixed - Initiative Interactions Summary 11/29/2018 Introduction

STRESS Handy Online Calendar Emails Instant Messages File Scout ... 11/29/2018 Introduction

Attentive User Interfaces Human attention and working memory are limited resources! 11/29/2018 Introduction

Attentive User Interfaces Goal of Attentive User Interfaces : To negotiate the volume and timing of human - computer communication in a natural way 11/29/2018 Introduction

Attentive User Interfaces enhance robustness and fluidity of human - computer collaboration importance of attentive cues goals intentions topics of interest signal offering and accepting a contribution in dialogues 11/29/2018 Introduction

Principles & Methodologies Overview Introduction Principles & Methodologies Applications Attention - Sensitive Messaging Mixed - Initiative Interactions Summary 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs sensing attention use perceptual sensors microphones cameras GPS accelerometers interaction with software devices online calendar history of user’s interests and activities 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs reasoning about attention model user’s attentive state reason about future attention and ideal attention-sensitive actions under uncertainty uses HMMs Bayesian Networks Influence Diagrams 11/29/2018 Principles & Methodologies

Principles & Methodologies Bayesian Networks abstract probabilisitic knowledge base reflecting causal knowledge graph with following qualities 1. random variables as nodes 2. directed links between the nodes 3. each node with conditional probability table 4. no directed cycles 11/29/2018 Principles & Methodologies

Principles & Methodologies Bayesian Networks Body Position Seminar- Room Head Pose Location Focus of Attention 11/29/2018 Principles & Methodologies

Principles & Methodologies Bayesian Networks Task: compute probability distribution for a set of query variables given value of some evidence variables P(Query | Evidence)  decisions about actions possible 11/29/2018 Principles & Methodologies

Principles & Methodologies Bayesian Networks kinds of inferences: Diagnostic - from effect to causes Causal - from cause to effects Intercausal - between causes of an effects Mixed - combinations of the ones above inference also possible over time using Dynamic Bayesian Networks 11/29/2018 Principles & Methodologies

Principles & Methodologies Influence Diagrams combines Bayesian Networks with nodes for action and utility 3 types of nodes : chance nodes : random variables action nodes : choice of action influencing following nodes utility nodes : computes the utility of actions given a state of the network 11/29/2018 Principles & Methodologies

Principles & Methodologies Influence Diagrams Body Position Value of Action Location Focus of Attention Expected Utility Actions Seminar- Room Head Pose Cost of Action 11/29/2018 Principles & Methodologies

Principles & Methodologies Influence Diagrams Algorithm 1. set evidence variable for current state 2. for all possible values of the action node do a) set action node to specific value b) calculate utility using standard probabilistic inference algorithm 3. return action with highest utility 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs graceful negotiation of turns and sense of acknowledgement of the request determine utility of interruption given the priority of the information signal information via nonintrusive channel sense user acknowledgement of the information 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs Communicating attention AUIs show their attention to user communicate focus of the user to other AUIs or remote people requesting user’s attention 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs Augmenting attentive resources presents information in estimated focus of user attenuating peripheral detail 11/29/2018 Principles & Methodologies

Principles & Methodologies Key Features of AUIs Sensing attention Reasoning about attention Negotiation of turns and sense of acknowledgement Communicating attention Augmenting attentive resources 11/29/2018 Principles & Methodologies

Overview Introduction Principles & Methodologies Applications Summary Attention - Sensitive Messaging Mixed - Initiative Interactions Summary 11/29/2018 Applications

The Notification Platform Project Microsoft Research, Redmond centered on economic based principles: value of information vs. attentive-sensitive cost of disruption 11/29/2018 Applications

The Notification Platform Project subscription of different sources and output devices possible considers user’s attention and location under uncertainty aware of fidelity and relevance of different communication channels 11/29/2018 Applications

The Notification Platform Project Emails Handy Online Calendar Notification Platform Instant Messages File Scout ... 11/29/2018 Applications

The Notification Platform Project Universal Inbox Bayesian attentive model Context Server & Context Whiteboard Decision model Profile of user’s preferences 11/29/2018 Applications

The Notification Platform Project 11/29/2018 Applications

The Notification Platform Project Example: A new message coming into the Universal Inbox 11/29/2018 Applications

The Notification Platform Project 1. How important is this message? - question of informing user now or later - computes cost of delayed review  Information Value 11/29/2018 Applications

The Notification Platform Project 2. Which device and modality should I use? a) What is the user doing now? Focus of Attention b) Will the user receive my notification on this device? c) Will I lose fidelity of information when using this device?  Transmitted Value Value of Action Device & Modality 11/29/2018 Applications

The Notification Platform Project 3. How expensive is it to alert the user now? - uses preferences stored in profile  Attentional cost Focus of Attention Cost of Action Device & Modality 11/29/2018 Applications

The Notification Platform Project for every device and modality compute the Expected Utility = Transmitted Value - Attentional cost  choose action with highest utility! Device & Modality Value of Action Cost of Action Expected Utility 11/29/2018 Introduction

The Notification Platform Project 11/29/2018 Applications

Overview Introduction Principles & Methodologies Applications Summary Attention - Sensitive Messaging Mixed - Initiative Interactions Summary 11/29/2018 Applications

DeepListener Project spoken language system address the “speech-target problem” use probabilistic models to compute the likelihood of being the target of speech, of what was said and of different intentions clarification dialogues use cues for conversational turn taking attentional lens or animated agent 11/29/2018 Applications

DeepListener Project 11/29/2018 Applications

Quartet Project continuous speech recognition system advanced model of attention: keyboard events visual pose disruption of attention between device and human natural language processing realises misrecognition and intention for someone else 11/29/2018 Applications

Quartet Project 11/29/2018 Applications

Overview Introduction Principles & Methodologies Applications Summary Attention - Sensitive Messaging Mixed - Initiative Interactions Summary 11/29/2018 Summary

Summary if we extend devices with attention sensors advanced representation & inferential methods communication capabilities 11/29/2018 Summary

Summary it is possible to build attentive sensible unintrusive sociable devices! but there is still much research to do : recognition, modelling, reasoning, psychology 11/29/2018 Summary

Literature E. Horvitz, C. M. Kadie, T. Paek, D. Hovel. Models of Attention in Computing and Communications: From Principles to Applications, Communications of the ACM 46(3):52-59, March 2003 Jeffrey S. Shell, Ted Selker, Roel Vertegaal: Interacting with groups of computers. Communications of the ACM 46(3): 40-46, March 2003 Paul P. Maglio, Christopher S. Campbell: Attentive agents. Communications of the ACM 46(3): 47-51, March 2003 Roel Vertegaal: Introduction. Communications of the ACM 46(3): 30-33, March 2003 http://research.microsoft.com/~horvitz/cacm-attention.htm Stuart Russell, Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 1995 11/29/2018 Summary

Thank you for your kind attention! 11/29/2018