1. 1 Sensing Techniques for Mobile Interaction Ken Hinckley Jeff Pierce Mike Sinclair Eric Horvitz Attentional User Interfaces Project Microsoft Research

2. 2 Mobile Devices vs. Mr. Cleo (An Informal Comparison) Mr. Cleo Aware of sounds, objects Knows if I walk in the room Selfish & Inconsiderate My Mobile Device Unaware of environment Oblivious to my presence Selfish & Inconsiderate How can we make Smart Computers?

3. 3 Sensitive Interfaces (a.k.a. “Not-So-Stupid Computing”) Computers are very fast idiots. They are oblivious to the external world. What is a “smart computer” anyway??? Can we build one? Do you really want one? Or one that is aware, respectful, … Not So Stupid? We’d be better off if computers were as dumb as my cat Can some very dumb sensors, with straightforward software, help address this problem? Design hardware/software interfaces that are sensitive to the user and the surrounding physical environment.

4. 4 Sensing for User Interaction Sense more than just explicit commands Simplify the interface using background information that is already there Point+Shoot Cameras One Button Many sensors

5. 5 Can “Background Sensing” Enhance Mobile UI? Real world has high cognitive / attentional demand Even “clicking a button” can be hard! Software ignorant of the changing physical context Device should adapt to current task / situation Naturally occurring “gestures” of use are missed Pick up, put down, look at, walk around with… Explosion of cheap, informative sensors On device or available via wireless network

6. 6 Tilt Sensor Proximity Range Sensor Touch Sensor Demos implemented as Windows CE applications Sensor I/O via PIC micro. VIDEO Mobile Sensor Prototype (Casio E105)

7. 7 Tilt Sensor Analog Devices 2-axis linear accelerometer Tilt relative to gravity But, other accelerations also affect signal Limitations: Cannot sense rotation around vertical axis Cannot tell up from down

8. 8 Activity Detection: Tilt Sensor Example Data PocketPC held at side Walking to elevator Looking at display Walking to meeting

9. 9 Voice Memo Detector Gesture allows general-purpose PIM to have special-purpose context of use Sensor Fusion: Must be holding device Tilt “like a phone” Hold close to face Audio is crucial to the interaction Relaxing stops recording

10. 10 Voice Memo: Workflow Analysis 1.Pick up device 2.Find the ¼” dia. button 3.Position hand to press button 4.Press & maintain tension 5.Listen for beep 6.Record your message 7.Release when done 8.Double-beep confirms 1.Pick up device (to face) 2.Listen for the beep 3.Record your message 4.Relax device when done 5.Double-beep confirms recording was made Normal approachSensor approach “I have to think about finding the button, pushing.it, holding it” “It was just listen for the beep”

11. 11 Sensed Memo Recording: Usability Issues “talk into it like a cell phone” is enough instruction to use “Quite a bit easier, I can focus” “Would use it more if it worked that way” 6/7 Ss preferred sensed gesture to button (4.3) Sensed gesture not easily discoverable “disorienting to put up to my face to talk” - 1 Ss False positives can occur: e.g. putting into sweater pocket But, button also has false positives…

12. 12 Tracking Experiment Not faster, but less demand on visual attention. User tracks “fly” on monitor, using mouse Records “Testing 123” (S, M) > C (p<.001) Sensed gesture may require less visual attention (p < 0.1)

13. 13 Portrait / Landscape Display Mode Detection “Snow Globe” Input controls rotated to match screen Easy (5.0/5.0). 6 Ss prefer tilting to menu; 1 Ss “I think it would drive me nuts” Sharing w/others Doesn’t interrupt

14. 14 Tilt vs. sensed orientation: Portrait / Landscape Detection: Implementation 5 display modes 2 Portrait 2 Landscape Flat Dead bands, 0.5s: keep screen stable Put-down problem: FIFO queue, look for stable orient.

15. 15 Tilt for Scrolling Touch screen bezel to clutch Easy to activate / maintain state Too easy to hit by accident (landscape mode) Sets “resting” orientation Hides on-screen UI (menu, taskbar) during scrolling Several transfer functions possible Rate ctrl: v = K * sgn(dA)  max(||dA|| - dA min, 0)  Single axis: only move along 1 primary axis at a time Dual axis: full panning in any direction Mixed axis: panning, but with affinity for primary axes

16. 16 Tilt: Scrolling & Then Some Contrast compensation Equalize “apparent contrast” Scrolling + Portrait/Landscape Don’t change P/L modes while scrolling! Don’t change when stop scrolling, either! Waits to see different orientation Not quite right; should switch after longer dwell (2-3 s?) User Testing: 5 Ss, compared to built-in direction pad “Good way to scroll the screen” – Agree (4.8 / 5.0) “I’d rather use the direction pad” – Disagree (1.8 / 5.0) One-handed operation, natural, simplifies the movement

17. 17 Power Management Automatic power-on Holding device in hand AND looking at display (flat in one axis, tilted fwd ~20 o in the other) for 0.5s Can’t power up in purse / pocket Won’t power up if you just touch it to push away Won’t power up if you just grab & hold at side (usually ) There is no auto-power-OFF feature, by design Best case: turns off & user doesn’t even notice feature Touch, tilt & proximity sensors prevent power-off or screen dimming while using device

18. 18 Other Fun Stuff Games Proximity For zooming – fun demo, but how to select? Proximal UI: Creepy “Hand of frustration” Shaking To switch applications (top app  bottom) Turn upside down & shake to erase :-) Impl. not robust enough to turn on all the time

19. 19 Conclusions New UI’s with better sensitivity to the user & the user’s physical environs Great potential to simplify & enhance the UI  new behaviors and services that users find compelling, useful, engaging, respectful Design, Implementation, & Usability challenges… Design must handle false positive / false negative cases Not a panacea. Only seems helpful for some tasks. Making things simpler vs. Loss of explicit control Promising area that needs more work

20. 20

21. 21 Proximity Sensor IR LED, 60 o, 40kHz IR receiver (used in TV’s) Gain  distance Some light sensitivity, e.g. sunlight

22. 22 Touch Sensor Mainly so we know when user is holding device Useful to disambiguate “intentional” gestures from accidental ones holding and tilting, vs. tilted while sitting in briefcase Also experimented with touch buttons around screen bezel

23. 23 Software Architecture “Context Server” whiteboard Shared memory to read variables Or, ask for Windows messages to notify of changes Apps can post any synthesized info back to server Some example context variables (events): Holding, Duration TiltAngleLR / FB DisplayOrientation Walking Proximity (z) ProximityState Scrolling VoiceMemoGesture

24. 24 Touch bezel for explicit control over tool bars Proximity of hand to screen? auxiliary UI appears when hand gets close Screen Real Estate Optimization