EyePhone: Activating Mobile Phones With Your Eyes Emiliano Miluzzo, Tianyu Wang, Andrew T. Campbell CS Department – Dartmouth College, Hanover, NH, USA

Miluzzo what is the future of sensing?

Miluzzo mobile phone sensing

Miluzzo mobile phone sensing

Miluzzo mobile phone sensing computationally capable (up to 2GHz CPU and 1 GB RAM)

Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing ….

Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing …. light sensor/camera

Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor mobile phone sensing …. light sensor/camera camera

Miluzzo front facing camera

Miluzzo how can we use it?

Miluzzo emotion assessment

Miluzzo emotion assessment in-car safety

Miluzzo emotion assessment human-phone interaction in-car safety

Miluzzo why HPI?

Miluzzo human-computer interaction - HCI because…

Miluzzo human-computer interaction - HCI human-phone interaction - HPI because…

Miluzzo human-computer interaction - HCI human-phone interaction - HPI because…

Miluzzo fairly ideal conditions for sensing array of sophisticated sensors subjects mostly static “sensing context” fairly constant subjects are on the move (video frames are blurred) limited sensing capabilities not ideal sensing context (dark, bright, etc.) hardware limitations HPIHCI

Miluzzo EyePhone

Miluzzo

Miluzzo mapping the position of the eye to an application icon on the phone display

Miluzzo mapping the position of the eye to an application icon on the phone display

Miluzzo mapping the position of the eye to an application icon on the phone display

Miluzzo the selected application (e.g., music) is activated with a wink

Miluzzo the selected application (e.g., music) is activated with a wink

Miluzzo EyePhone design

Miluzzo eye template creation

Miluzzo eye template creation offline for different environments

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair running on pcrunning on the N810

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking on every frame

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame wink detection

Miluzzo eye template creation offline for different environments image differencing over a set of frames capturing a blink. The result is a binary image with the eye pair eye tracking based on a correlation function operating over the current frame and the template on every frame wink detection based on the correlation function values used in the tracking phase

Miluzzo evaluation

Miluzzo C++ & OpenCV implemented on the Nokia Tablet 810

Miluzzo experimental results

Miluzzo accuracy steady, daylight

Miluzzo accuracy walking, daylight

Miluzzo accuracy Vs distance from display

Miluzzo accuracy Vs distance from display

Miluzzo possible applications?

Miluzzo beyond touch screen…

Miluzzo let’s recap…

Miluzzo let’s recap… smartphone’s computation capabilities are growing

Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context)

Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context) need for eye tracking techniques that are robust against context, mobility, distance from display

Miluzzo let’s recap… smartphone’s computation capabilities are growing smarter open-eye template creation policies are needed (to be robust across different context) smarter open-eye template creation policies are needed (to be robust across different context) need for eye tracking techniques that are robust against context, mobility, distance from display more efficient false eye pair and “false” winks filtering to avoid undesired app launches

Miluzzo in brief…

Miluzzo in brief… mobile phone sensing and HPI technology are the future!

Thank you! Miluzzo Mobile Sensing Group