1. EyePhone: Activating Mobile Phones With Your Eyes Emiliano Miluzzo, Tianyu Wang, Andrew T. Campbell CS Department – Dartmouth College, Hanover, NH, USA

2. miluzzo@cs.dartmouth.eduEmiliano Miluzzo what is the future of sensing?

3. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mobile phone sensing

4. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mobile phone sensing

5. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mobile phone sensing computationally capable (up to 2GHz CPU and 1 GB RAM)

6. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing ….

7. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing …. light sensor/camera

8. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing …. light sensor/camera camera

9. miluzzo@cs.dartmouth.eduEmiliano Miluzzo front facing camera

10. miluzzo@cs.dartmouth.eduEmiliano Miluzzo how can we use it?

11. miluzzo@cs.dartmouth.eduEmiliano Miluzzo emotion assessment

12. miluzzo@cs.dartmouth.eduEmiliano Miluzzo emotion assessment in-car safety

13. miluzzo@cs.dartmouth.eduEmiliano Miluzzo emotion assessment human-phone interaction in-car safety

14. miluzzo@cs.dartmouth.eduEmiliano Miluzzo why HPI?

15. miluzzo@cs.dartmouth.eduEmiliano Miluzzo human-computer interaction - HCI because…

16. miluzzo@cs.dartmouth.eduEmiliano Miluzzo human-computer interaction - HCI human-phone interaction - HPI because…

17. miluzzo@cs.dartmouth.eduEmiliano Miluzzo human-computer interaction - HCI human-phone interaction - HPI because…

18. miluzzo@cs.dartmouth.eduEmiliano Miluzzo fairly ideal conditions for sensing array of sophisticated sensors subjects mostly static “sensing context” fairly constant subjects are on the move (video frames are blurred) limited sensing capabilities not ideal sensing context (dark, bright, etc.) hardware limitations HPIHCI

19. miluzzo@cs.dartmouth.eduEmiliano Miluzzo EyePhone

20. miluzzo@cs.dartmouth.eduEmiliano Miluzzo

21. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mapping the position of the eye to an application icon on the phone display

22. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mapping the position of the eye to an application icon on the phone display

23. miluzzo@cs.dartmouth.eduEmiliano Miluzzo mapping the position of the eye to an application icon on the phone display

24. miluzzo@cs.dartmouth.eduEmiliano Miluzzo the selected application (e.g., music) is activated with a wink

25. miluzzo@cs.dartmouth.eduEmiliano Miluzzo the selected application (e.g., music) is activated with a wink

26. miluzzo@cs.dartmouth.eduEmiliano Miluzzo EyePhone design

27. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation

28. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments

29. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair

30. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair running on pcrunning on the N810

31. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking

32. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking on every frame

33. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame

34. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame

35. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame wink detection

36. miluzzo@cs.dartmouth.eduEmiliano Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame wink detection based on the correlation function values used in the tracking phase

37. miluzzo@cs.dartmouth.eduEmiliano Miluzzo evaluation

38. miluzzo@cs.dartmouth.eduEmiliano Miluzzo C++ & OpenCV implemented on the Nokia Tablet 810

39. miluzzo@cs.dartmouth.eduEmiliano Miluzzo experimental results

40. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accuracy steady, daylight

41. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accuracy walking, daylight

42. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accuracy Vs distance from display

43. miluzzo@cs.dartmouth.eduEmiliano Miluzzo accuracy Vs distance from display

44. miluzzo@cs.dartmouth.eduEmiliano Miluzzo possible applications?

45. miluzzo@cs.dartmouth.eduEmiliano Miluzzo beyond touch screen…

46. miluzzo@cs.dartmouth.eduEmiliano Miluzzo let’s recap…

47. miluzzo@cs.dartmouth.eduEmiliano Miluzzo let’s recap… smartphone’s computation capabilities are growing

48. miluzzo@cs.dartmouth.eduEmiliano Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context)

49. miluzzo@cs.dartmouth.eduEmiliano Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context) need for eye tracking techniques that are robust against context, mobility, distance from display

50. miluzzo@cs.dartmouth.eduEmiliano Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context) need for eye tracking techniques that are robust against context, mobility, distance from display more efficient false eye pair and “false” winks filtering to avoid undesired app launches

51. miluzzo@cs.dartmouth.eduEmiliano Miluzzo in brief…

52. miluzzo@cs.dartmouth.eduEmiliano Miluzzo in brief… mobile phone sensing and HPI technology are the future!

53. Thank you! miluzzo@cs.dartmouth.eduEmiliano Miluzzo Mobile Sensing Group http://sensorlab.cs.dartmouth.edu