Automated Reading Assistance System Using Point-of-Gaze Estimation M.A.Sc. Thesis Presentation Automated Reading Assistance System Using Point-of-Gaze Estimation Jeffrey J. Kang Supervisor: Dr. Moshe Eizenman Department of Electrical and Computer Engineering Institute of Biomaterials and Biomedical Engineering January 24, 2006

Introduction Reading Visual examination of text Convert words to sounds to activate word recognition We learn appropriate conversions through repetitive exposure to word-to-sound mappings Insufficient reader skill or irregular spelling can lead to failed conversion: assistance is required Objective: Develop an automated reading assistance system that automatically vocalizes unknown words in real-time on the reader’s behalf. The system should operate within a natural reading setting.

What We Need To Do — Step 1 1. Identify the word being read, in real-time 2. Detect when the word being read is an unknown word 3. Vocalization of the unknown word

Identifying the Word Being Read Identify the viewed word using point-of-gaze estimation Point-of-gaze is: Where we are looking with the highest visual acuity region of the retina Intersection of the visual axis of each eye within the 3D scene Intersection of the visual axis one eye with a 2D plane

Point-of-Gaze Estimation Methodologies 1. Head-mounted 2. Remote (no head-worn components)

Head-mounted Point-of-Gaze Estimation Based on principle of tracking the pupil centre, and corneal reflections to measure eye position Point-of-gaze is estimated with respect to a coordinate system attached to the head scene camera eye camera IR LEDs hot mirror corneal reflections pupil centre

Point-of-Gaze in Head Coordinate System Point-of-gaze is measured in the head coordinate system, and placed on the scene camera image

Locating the Reading Object The position of the reading object is determined by tracking markers

Mapping the Point-of-Gaze Establish point correspondences from the estimated positions of the markers in the scene image the known positions of the markers on the reading object Homographic mapping of point-of-gaze from scene camera image to reading object coordinate system

Identify the Reading Object Extract the barcode from the scene camera image to identify the reading object (e.g. page number) Match barcode to database of reading objects to determine what text is being read

Identifying the Word Being Read Using the mapped point-of-gaze, identify the word being read by table lookup

Sample Reading Video

Mapping Accuracy

Point-of-Gaze Estimation Methodologies 1. Head-mounted 2. Remote (no head-worn components)

O 2D scene object Z X P C visual axis Y Remote Point-of-Gaze Estimation Point-of-gaze is estimated to a fixed coordinate system C – centre of corneal curvature P – point-of-gaze IR LEDs eye camera computer screen

O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Moving Reading Card How can point-of-gaze be estimated to a coordinate system attached to a moving reading object?

O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Estimate Motion R, T t0t0 t1t1

O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Use a Scene Camera and Targets t0t0 t1t1 Scene Camera

H 0 O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Calculate Two Homographies t0t0 t1t1 H1H1 Scene Camera

O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Decompose Homography Matrices t0t0 t1t1 Scene Camera R 0, T 0 R 1, T 1

O assumed position of 2D scene object Z X P C visual axis P’ true position of 2D scene object Y Calculate Motion of 2D Scene Object t0t0 t1t1 Scene Camera R, T R 0, T 0 R 1, T 1

Point-of-Gaze Accuracy

What We Need To Do: Step 2 1. Identify the word being read, in real-time 2. Detect when the word being read is an unknown word 3. Vocalization of the unknown word

Dual Route Reading Model Coltheart, M. et al. (2001)

Each word’s graphemes are processed in parallel Dual Route Reading Model

Each word’s graphemes are individually converted into phonemes based on mapping rules

Detecting Unknown Words For unknown words, the lexical route fails and the slower non-lexical route is used Hypothesis: we can differentiate between known and unknown words by the duration of the processing time

Processing Time

Setting a Threshold Curve

Threshold curve is a function of word length Model processing time for known words (length k ) as a Gaussian random variable  (μ k, σ k 2 ) Estimate μ k, σ k 2 from a short training set for each subject Each point on threshold curve is given by α is the constrained probability of false alarm Setting the Threshold

Experiment: Detecting Unknown Words Remote point-of-gaze estimation system Reading material presented on computer screen Head position stabilized using a chinrest Four subjects read from 40 passages of text 20 passages aloud and 20 passages silently Divided into training set to “learn” μ k, σ k 2 and set detection threshold curves Set false alarm probability α = 0.10 Evaluate detection performance

Experiment: Detecting Unknown Words

Experiment: Natural Setting Reading Assistance Natural reading pose Unrestricted head movement Reading material is hand-held Head-mounted eye-tracker Identify viewed word in real-time Measure per-word processing time Detecting unknown words Processing time threshold curves established in previous experiment Assistance Detection of unknown word activates vocalization

Experiment: Natural Setting Reading Assistance Results Point-of-gaze mapping method accommodated head and reading material movement without reducing detection performance SubjectDetection RateFalse Alarm Rate M.E P.L

Conclusions Developed methods to map point-of-gaze estimates to an object coordinate system attached to a moving 2D scene object (e.g. reading card) Head-mounted system Remote system Developed method to detect when a reader encounters an unknown word Demonstrated principle of operation for an automated reading assistance system

Future Work Implement reading assistant using remote-gaze estimation methodology Validate efficacy of system as a teaching tool for unskilled English readers, in collaboration with an audiologist Evaluate other forms of assistive intervention e.g. translation, definition

Questions?