1. Forearm Electromyography Muscle-Computer Interfaces Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-Computer Interfaces T. Scott Saponas University of Washington Desney S. Tan Microsoft Research Dan Morris Microsoft Research Ravin Balakrishnan University of Toronto

2. Physical Transducers Leverage Human Expertise 2

3. Need for Hands Free Input 3

4. Advances in Muscle Sensing Enable Muscle-Computer Interfaces 4 …

5. 5 Muscles Activate via Electrical Signal

6. 6 Electrical Signal can be sensed by Electromyography (EMG)

7. EMG for Diagnostics, Prosthetics & HCI 7 Jacobsen, et al. “Utah Arm”

8. EMG for Diagnostics, Prosthetics & HCI 8 Jacobsen, et al. “Utah Arm” Costanza, et al. “Intimate interfaces in action”

9. EMG for Diagnostics, Prosthetics & HCI 9 Jacobsen, et al. “Utah Arm” Costanza, et al. “Intimate interfaces in action” Naik, et al. “Hand gestures”

10. EMG for Diagnostics, Prosthetics & HCI 10 Jacobsen, et al. “Utah Arm” Costanza, et al. “Intimate interfaces in action” Wheeler & Jorgensen “Neuroelectric joysticks” Naik, et al. “Hand gestures”

11. Detecting Finger Gestures Challenging 11

12. Sensors Placed on Upper Forearm 12

13. Sensors Placed on Upper Forearm 13

14. Gesture Sets 14

15. Gesture Sets 15

16. Lift 16

17. Tap 17

18. Position 18

19. Pressure 19

20. 20 Pressure Position Tap Lift

21. 21 250 millisecond sample X 8 Sensors Features Support Vector Machine training data user model test data evaluation machine learning Gesture Classification Technique

22. 22 250 millisecond sample Root Mean Square (RMS) 28 ratios between channels Frequency Energy 10 Hz bins Phase Coherence 28 ratios between channels X 8 Sensors Features Support Vector Machine training data user model test data evaluation machine learning

23. Randomized Block Design 23 1234 random delay

24. Randomized Block Design 24 12341234 X 50 random delay random order

25. 12 participants aged 20 – 63 years (mean 46) 8 female; 4 male daily computer users right-handed 90 minutes 25

26. Ten-Fold Cross-Validation 26 Lift

27. Ten-Fold Cross-Validation 27 Tap

28. Ten-Fold Cross-Validation 28 Position

29. Ten-Fold Cross-Validation 29 Pressure

30. How much training data? 30

31. What are we really measuring? Skin moving over muscle creates noise Distant muscle contractions Gestures are complex movements 31

32. Limitations of Current Evaluation Works best for SINGLE user SINGLE session Offline Analysis Approximation of sensor armband 32

33. Forearm Electromyography for Muscle-Computer Interfaces Demonstrated possibility of gesture sets using pressure, position, & all five fingers 33 Future: Wireless & dry sensors Dense auto-configurable band Cross-user models Quick compound gestures

34. Interaction Possibilities Virtual keyboards Hands busy controls 3D gestural interaction Eye-free mobile interaction 34

35. thanks! acknowledgements: Sumit Basu, James Fogarty, Jon Froehlich, Kayur Patel, Meredith Skeels and our study participants 35 T. Scott Saponas University of Washington Desney S. Tan Microsoft Research Dan Morris Microsoft Research Ravin Balakrishnan University of Toronto http://research.microsoft.com/users/dan/muci/ …

36. 36

37. 37 Pressure Position Tap Lift

38. Labeling Training Data With Best Data 38 stimulus stimulus labelrest label

39. Single Sample Classification 39 stimulus ? ? ? stimulus thumb 1 index 5 middle 1 winner Whole Trial Classification

40. Collect Pilot Data, Develop Classification Techniques, Evaluation 40 collect pilot datadevelop classification techniques define gesture sets collect test data offline analysis experiment