1. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu The Effectiveness of Haptic Cues as an Assistive Technology for Human Memory

2. Human Memory Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

3. Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

4. Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

5. Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

6. Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

7. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Memory Impairments Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

8. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Visual and Audio Cues Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu DeVaul, R.W., Pentland, A. and Corey, V.R. The memory glasses: subliminal vs. overt memory support with imperfect information. (2005) Lin, W. and Hauptmann A.G.A Wearable Digital Library of Personal Conversations. (2002) Disruptive Prominent

9. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Vibro-Tactile Cues Unobtrusive Discreet

10. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Vibro-Tactile Interaction Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

11. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Hypothesis Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Vibro-tactile cues can improve human memory recall and recognition

12. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Priming Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Effective cross-modal priming for visual and haptic stimuli Ballesteros S., Reales J.M., Mayas J. and Heller M.A. Selective Attention modulates visual and haptic repetition priming: effects in aging and Alzheimer's disease. (2008)

13. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Attention and Navigation Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Haptic feedback aids attention and navigation Young, J.J., Tan, H.Z., Gray, R. Validity of haptic cues and its effect on priming visual spatial attention. (2003) Kildal, J. and Brewster, S.A. VibroTactile External Memory Aids in Non-Visual Browsing of Tabular Data. (2006)

14. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Multimodal Priming Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Retrieval of audio and video information is assisted with vibro-tactile cues

15. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

16. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

17. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

18. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

19. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

20. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Wearable Haptic Feedback Device Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

21. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Cue 1: two medium pulsesCue 2: strong oscillating pulses Cue 3: short soft pulse Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Example Cues

22. Training Phase Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Procedure Training Phase Pause Test Phase Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Pause

23. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Method Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

24. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Four User Studies “Blending” Audio Recognition Visual Recognition Free Recall Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

25. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Blending 24 participants, 15 male, 9 female, ages 21-65 Shape Recognition Test Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

26. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

27. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Auditory Recognition 20 participants: 12 male, 8 female, ages 21-65 Mandarin word translation test Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

28. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

29. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Visual Recognition 22 participants: 16 male, 6 female, ages 21-65 Face memory test Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

30. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Visual Recognition Test Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

31. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Free Recall 24 participants: 16 male, 8 female, ages 21-65 Ebbinghaus nonsense syllable test (“hik”, “lup”) Ebbinghaus, H. Memory: A Contribution to Experimental Psychology. Teacher’s College (1913). Dover Publications. Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

32. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

33. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Results Audio recognition Shape recognition Face recognition Free recall +11% (5 trials, p = 0.075) no effect -20% (7 trials, p = 0.009) Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

34. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

35. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

36. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

37. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Haptic Cues and Free Recall Significantly lower performance with cues Declarative vs. Perceptual memory Our cues did not have semantic meaning Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

38. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Haptic Cues and Cognitive Load Cues were ‘distracting’ Perceptual differences between users Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

39. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Haptic Cues as a Memory Aid Memory aid for people with poor or impaired memory Memory enhancing treatments often target low performers Aldenkamp, A. P., et al., Effect of Sabeluzole (R 58 735) on Memory Functions in Patients with Epilepsy. (1995) Mattay, V. S. et. al.. Effects of Dextroamphetamine on Cognitive Performance and Cortical Activation. (2000) Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

40. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Opportunities Cue vocabulary Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Maximum number of cues Effectiveness of cues for people with functional memory impairments

41. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Pulse Duration Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Pulse 1Pulse 2

42. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Pulse Intensity Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Pulse 1Pulse 2

43. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Delay Between Pulses Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Pulse 1Pulse 2

44. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Cue Vocabulary Study 14 Participants, age 18-45 Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

45. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Ratings of Pulse Duration Differences Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

46. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Ratings of Pulse Intensity Differences Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

47. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Ratings of Differences in Delay Between Pulses Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

48. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Ongoing Work Systematically pick cues from the sampled space Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Determine the maximum number of cues that can be remembered Recruit subjects with functional memory impairments

49. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Future Work Adjust cue intensity to individual’s perception Strive for context-awareness Add semantic meaning to cues Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

50. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon The Effectiveness of Haptic Memory Cues Haptic cues as a memory aid for poor performers Assistive technology for memory-impaired users Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

51. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Vibro-Tactile Devices

52. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu

53. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Roberta Klatzky NSF Intel Thank you Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu http://isiria.files.wordpress.com/2008/04/foreign_language.jpg http://internetservices.readingeagle.com/blog/gavel/archives/cellphone.jpg http://www.blendfilms.com/movies/handshake_small.mov http://spiddlement.files.wordpress.com/2008/05/todolist.jpg http://static.howstuffworks.com/gif/brain-1.jpg http://www.sytechcorp.com/new_site/SytechCorp/SyTechCorpXY/images/Pager_AA021263.jpg Image Credits

54. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Questions Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Kuznetsov, S., Dey, A. K., Hudson, S. E. The Effectiveness of Haptic Cues as a Ubiquitous Memory Aid. Pervasive Computing, 2009

55. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Sixth Sense, MIT Media Lab, 2009

56. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Point localization thresholds for males for different areas of the body (Weinstein, 1968)

57. Living Environments Lab Human- Computer Interaction Institute Carnegie Mellon Questions Stacey Kuznetsov, Anind K. Dey, Scott E. Hudson, Eric Paulos {stace, anind, hudson, paulos} @ cs.cmu.edu Kuznetsov, S., Dey, A. K., Hudson, S. E. The Effectiveness of Haptic Cues as a Ubiquitous Memory Aid. Pervasive Computing, 2009