Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski

 The Project ◦ Why is it important  The Goals ◦ System breakdown  Image recognition  Point transformation  User Interface  The Results

What is our project?

 Track a user’s eye and use the information to control a computer cursor

 Enhances Human Computer Interaction ◦ Speed of use ◦ Hands-free use

 3 Part System ◦ Image Processing Application ◦ Calibration and Mapping system ◦ GUI designed for gaze-based interaction  Systems developed concurrently and independently  Separate Applications at run-time

Hardware and Image Processing Application

 Hardware ◦ Camera  QuickCam Pro for Notebooks  Visible Spectrum Camera ◦ Polarizer  Tiffen 25mm polarizing filter  Removes glare from eye reflections ◦ Lighting  Diffuse LEDs  Slightly distracting to the user, but necessary to provide light for the camera

 LitEye LE-500 ◦ High resolution (SVGA) ◦ Color Display ◦ Translucent or opaque operation ◦ Stationary relative to user’s eye

 Real time pupil tracking system ◦ Developed in C using OpenCV image processing libraries ◦ Traditional image processing and blob tracking  Capabilities ◦ Locate and determine center of pupil in image ◦ Low light and high reflection environments ◦ All eye colors ◦ Data logging and static test modes ◦ Packaged into self contained Windows installer for easy deployment onto any computer

 The Process ◦ Capture Image ◦ Extract Red Channel ◦ Smooth image ◦ Apply binary threshold ◦ Locate blobs ◦ Reject false positives ◦ Determine center of pupil blob ◦ Adapt threshold ◦ Repeat

 Summary ◦ The Good  Dynamically adapts to changing lighting conditions and eye types  Maintains performance in low-light and specularly noisy conditions ◦ The Bad  Still relies on Logitech camera drivers  Extreme reflections still cause problems

Raw CapturePre-Processed Image Completed Recognition

 Examples of performance in poor conditions Low Light Difficult False Positive

Calibration and point mapping

 System for mapping the location of the center of the pupil to a pixel on a computer screen  Reasons ◦ Geometry  The eye is not flat but a screen is not ◦ User Customization  All eyes are different  Everyone wears the HMD differently ◦ User Training  Calibration system also acts as a quick tutorial

 3 dimensional best fit plane ◦ Currently using a 4 th degree best fit X pix = A 1 + X eye *B 1 + Y eye *C 1 Y pix = A 2 + X eye *B 2 + Y eye *C 2  Calibration sub-system determines these coefficients

 How do we solve the problem? ◦ Multiple Variable Linear Regression – Least Squares Y = B 0 + B 1 x 1 + … + B k x k ◦ Uses matrix algebra to obtain a coefficient matrix B[] = (X’X) -1 X’Y

 Results ◦ pixelX = *eyeX – 2.9*eyeY ◦ pixelY = *eyeX – 5.6*eyeY B[] = (X’X) -1 X’Y

 How do we click? ◦ Monitor eye movements ◦ Identify pauses ◦ When eye position is within a small percentage for a certain amount of time ◦ Currently set at 5 frames (~200 mS) ◦ Generally, it takes 230 mS for a hand to click a mouse.

Communications and Custom GUI interface

 Why is it important? ◦ Allows multiple processes to communicate ◦ Allows relay of time sensitive information  UDP vs. TCP ◦ UDP -> ‘Connectionless’ ◦ TCP -> ‘Connection oriented’  Multi-threading ◦ Necessary for running multiple pieces of code in a single process

Multi-threading issues present

 Multi-threading ◦ Issues with public variable usage ◦ Solutions we are using:  Critical Section  Raises thread priority (does not allow for interrupts) Receive request for data (over UDP) Raise Thread priority Lower Thread Priority Reply to request Read Variable

 Custom GUI for Gaze Tracking Applications  Why? ◦ Gaze tracking accuracy limited by inherent properties of human vision system ◦ Traditional GUI too small and intrusive for use with transparent HMD ◦ Demonstrate applications of gaze tracking

 Achieved via a combination of Hardware and Software  Software: WPF & C# ◦ XAML (eXtensible Application Markup Language) ◦ Very similar to XML ◦ Uses ‘code-behind’ in a similar style to javascript  ‘code-behind’ is in C#

 Multiple applications have been created within the interface  Multiple screens for functionality testing ◦ even games  Additional ability to minimize interface and interact with external applications is present

Calibration Screen Main Screen

What did our test results show?

Who helped us out?

 Northrop Grumman  LitEye  Dr. Malinowski and the EE faculty  Mr. Mattus & Mr. Schmidt