The Case for Operating System Management of User Attention Kyungmin Lee, Jason Flinn, and Brian Noble University of Michigan

Trend in mobile app interaction Using apps while performing primary tasks Apps initiate interactions Kyungmin Lee

Interaction in various user contexts User’s current primary activity ✖ ✖ ? ✔ Application is unaware of user’s context! Kyungmin Lee

Existing solution: Let user decide Set policy for each app Disable all interactions Too coarse grained! All or nothing Kyungmin Lee

Our proposed approach Mobile OS Mobile sensors Extract user’s context Interactions Deliver now Modify format Defer Interactions Kyungmin Lee

Outline Motivation Our vision Our proposed approach Challenges Kyungmin Lee

Our vision Do not interrupt! User’s current context Interrupt? Can you pick up milk? From: Your wife User’s current context Interrupt? Do not interrupt! Kyungmin Lee

Interrupt! via audio interaction Our vision Dangerous road conditions ahead User’s current context Interrupt? Interrupt! via audio interaction Kyungmin Lee

Our vision Interrupt! User’s current context Interrupt? Can you pick up milk? From: Your wife User’s current context Interrupt? Interrupt! Kyungmin Lee

Manage user attention as a resource 100% <Priority level> It’s a scheduling problem! User’s activity Attention level 100% Visual Auditory Cognitive Haptic Attention level 100% <Priority level> Visual Auditory Cognitive Haptic Interaction Attention demand Visual Auditory Cognitive Haptic Kyungmin Lee

Our proposed approach Priority Attention level Attention level Very low Low Medium High Very high 100% Attention level User’s current context Visual Auditory Cognitive Haptic Kyungmin Lee

Our proposed approach Priority Attention demand Attention demand Very low Low Medium High Very high 100% Can you pick up milk? From: Your wife Interrupt? Attention demand Visual Auditory Cognitive Haptic Kyungmin Lee

Our proposed approach No delivery! Attention level after delivery Medium priority Can you pick up milk? From: Your wife 100% Attention level High priority Visual Auditory Cognitive Haptic No delivery!

Our proposed approach Attention level after delivery Very high priority Dangerous road conditions ahead 100% Attention level High priority Visual Auditory Cognitive Haptic

Our proposed approach Deliver! Attention level after delivery Change to audio modality Very high priority Dangerous road conditions ahead 100% Attention level High priority Visual Auditory Cognitive Haptic Deliver!

Our proposed approach Deliver! Attention level after delivery Cognitive attn. load has dropped Medium priority Can you pick up milk? From: Your wife 100% Attention level High priority Visual Auditory Cognitive Haptic Deliver!

Challenges in determining priority Med. priority High priority From: A colleague From: A colleague High priority Low priority Friend’s request Friend’s request Kyungmin Lee

Learn from user’s behavior High priority Low priority Kyungmin Lee

Interaction’s attention demand Extend AMC (Mobisys ‘13) Button size ✔ Button closeness Text contrast ratio Word count ✖ Animation Scrolling Kyungmin Lee

Interaction’s attention demand Extend AMC (Mobisys ‘13) Attention demand Demand level Visual Auditory Cognitive Kyungmin Lee

Estimating user’s attention level Very high priority Same activity, but different priority level 100% Highly engaged activity Attention level Low priority Visual Auditory Cognitive Haptic Lowly engaged activity Kyungmin Lee

Conclusion Our vision: Right interaction at the right time Our proposed approach Treat user attention as a shared resource Determine priorities of interaction and activity Consider Attention level supply vs. demand Kyungmin Lee

Questions? Kyungmin Lee