Eyes Alive Sooha Park - Lee Jeremy B. Badler - Norman I. Badler University of Pennsylvania - The Smith-Kettlewell Eye Research Institute Presentation Prepared By: Chris Widmer CSE 4280

Outline Introduction Introduction Motivation Motivation Background Background Overview of System Overview of System Descriptions Descriptions Results Results Conclusions Conclusions

Introduction Eye movement an important expression technique Eye movement an important expression technique Statistical eye movement model based on empirical data Statistical eye movement model based on empirical data

Motivation Natural look eye movement for animations of close-up face views Natural look eye movement for animations of close-up face views Traditionally difficult to attain accurate eye movement in animations Traditionally difficult to attain accurate eye movement in animations No proposals for saccadic eye movement for easy use in speaking/expressive faces No proposals for saccadic eye movement for easy use in speaking/expressive faces Recent interest in construction of human facial models Recent interest in construction of human facial models

Background To build a realistic face model: To build a realistic face model: –Geometry modeling –Muscle behavior –Lip synchronization –Text synthesis Research has traditionally not focused on eye movement. Research has traditionally not focused on eye movement.

Background Eyes are essential for non-verbal communication Eyes are essential for non-verbal communication –Regulate flow of conversation –Search for feedback –Express emotion –Influence of behavior New approach based on statistical data and empirical studies

Saccades Rapid movements of both eyes from one gaze position to another. Rapid movements of both eyes from one gaze position to another. –Only Eye Movement Executed Consciously –Balance conflicting demands of speed and accuracy Magnitude – angle the eyeball rotates to change position Magnitude – angle the eyeball rotates to change position Direction – 2D axis of rotation, 0 degrees to the right Direction – 2D axis of rotation, 0 degrees to the right Duration – Time of movement Duration – Time of movement Inter-saccadic Interval – time between saccades Inter-saccadic Interval – time between saccades

Saccades Example: Magnitude 10, 45 degrees Example: Magnitude 10, 45 degrees –Rotating 10 degrees, right-upward Initial/Final Acceleration: 30,000 deg/sec Initial/Final Acceleration: 30,000 deg/sec Peak Velocity – 400 – 600 deg/sec Peak Velocity – 400 – 600 deg/sec Reaction Time: 180 – 220 msec Reaction Time: 180 – 220 msec Duration and Velocity Functions of Magnitude Duration and Velocity Functions of Magnitude Magnitude Approximation Magnitude Approximation D = D + d * A D = D 0 + d * A D = Duration, A = Amplitude, d = increment in duration per degree (2-2.7 msec/deg), D = Intercept (20-30 ms) D = Duration, A = Amplitude, d = increment in duration per degree (2-2.7 msec/deg), D 0 = Intercept (20-30 ms) Often accompanied by head rotation Often accompanied by head rotation

Background Three Functions of Gaze Three Functions of Gaze –Sending Social Signals –Open Channel to Receive Information –Regulate Flow of Conversation

Overview of System 1. Eye-tracking images analyzed, statistically based model generated using Matlab 2. Lip movements/Eye Blinks/Head Rotation analyzed by alterEGO face motion analysis system

Overview of System Face Animation Parameter (FAP) File Face Animation Parameter (FAP) File Eye Movement Synthesis System (EMSS) Eye Movement Synthesis System (EMSS) –Adds eye movement data to FAP file Modified from face2face’s animator plug-in for 3D Studio Max Modified from face2face’s animator plug-in for 3D Studio Max

Analysis of Data Eye movements recorded with eye-tracking visor (ISCAN) – monocle and two miniature cameras Eye movements recorded with eye-tracking visor (ISCAN) – monocle and two miniature cameras One views environment from left eye perspective, other is close-up of left eye One views environment from left eye perspective, other is close-up of left eye Eye image recorded Eye image recorded Device tracks by comparing corneal reflection of the light source relative to the location of the pupil center Device tracks by comparing corneal reflection of the light source relative to the location of the pupil center –Reflection acts as reference point while pupil changes during movement

Analysis of Data Pupil position found using pattern mapping Pupil position found using pattern mapping Default threshold grey level using Canny Edge Detection operator Default threshold grey level using Canny Edge Detection operator Positional histograms along X and Y axis calculated Positional histograms along X and Y axis calculated Two center points with maximum correlation chosen Two center points with maximum correlation chosen

Analysis of Data

Saccade Magnitude Saccade Magnitude Frequency of a specific magnitude (least mean squares distribution) Frequency of a specific magnitude (least mean squares distribution) d = Distance traversed by pupil center r = radius of eyeball (1/2 of x max P = % chance to occur A = Saccade Magnitude (Degrees)

Analysis of Data Saccade Duration measured with 40 deg/sec threshold Saccade Duration measured with 40 deg/sec threshold Used to derive instantaneous velocity curve for every saccade Used to derive instantaneous velocity curve for every saccade Duration of each movement normalized to 6 frames Duration of each movement normalized to 6 frames Two classes of Gaze: Two classes of Gaze: –Mutual –Away

Talking vs. Listening

Synthesis of Eye Movement Attention Monitor (AttMon) Attention Monitor (AttMon) Parameter Generator (ParGen) Parameter Generator (ParGen) Saccade Synthesizer (SacSyn) Saccade Synthesizer (SacSyn)

Head Rotation Monitoring

Synthesis of Natural Eye Movement AttMon determines mode, changes in head rotation, gaze state AttMon determines mode, changes in head rotation, gaze state ParGen determines saccade magnitude, direction, duration, and instantaneous velocity ParGen determines saccade magnitude, direction, duration, and instantaneous velocity SacSyn synthesizes and codes movements into FAP values SacSyn synthesizes and codes movements into FAP values

Synthesis of Natural Eye Movement Magnitude determined by inverse of fitting function shown earlier (Slide 16) Magnitude determined by inverse of fitting function shown earlier (Slide 16) Mapping guarantees same probability distribution as empirical data Mapping guarantees same probability distribution as empirical data Direction determined by head rotation (threshold), and distribution table Direction determined by head rotation (threshold), and distribution table –Uniform Distribution, 0 to 100 –8 non-uniform intervals assigned to respective directions

Synthesis of Natural Eye Movement Duration determined by first equation, respective values for d and D Duration determined by first equation, respective values for d and D 0 Velocity determined by using fitted instantaneous velocity curve Velocity determined by using fitted instantaneous velocity curve SacSyn system calculates sequence of coordinates for sys centers SacSyn system calculates sequence of coordinates for sys centers Translated into FAP values, rendered in 3D Studio MAX Translated into FAP values, rendered in 3D Studio MAX Face2face animation plug-in to render animations with correct parameters Face2face animation plug-in to render animations with correct parameters

Results 3 Different Methods Tested 3 Different Methods Tested Type 1 -> No Saccadic Movements Type 1 -> No Saccadic Movements Type 2 -> Random Eye Movement Type 2 -> Random Eye Movement Type 3 -> Sampled from Estimated Distributions (synchronized with head movements) Type 3 -> Sampled from Estimated Distributions (synchronized with head movements) Tests were subjective Tests were subjective

Results Q1: Did the character on the screen appear interested in (5) or indifferent (1) to you? Q1: Did the character on the screen appear interested in (5) or indifferent (1) to you? Q2: Did the character appear engaged (5) or (1) distracted during the conversation? Q2: Did the character appear engaged (5) or (1) distracted during the conversation? Q3: Did the personality of the character look friendly (5) or not (1)? Q3: Did the personality of the character look friendly (5) or not (1)? Q4: Did the face of the character look lively (5) or deadpan (1)? Q4: Did the face of the character look lively (5) or deadpan (1)? Q5: In general, how would you describe the character? Q5: In general, how would you describe the character?

Results

Conclusions Saccade Model for Talking and Listening Modes Saccade Model for Talking and Listening Modes 3 Different Eye Movements: Stationary, Random, Model-based 3 Different Eye Movements: Stationary, Random, Model-based Model-based scored significantly higher Model-based scored significantly higher Eye Tracking Data recorded from a subject Eye Tracking Data recorded from a subject –New recorded data for every character This model allows any number of unique eye movement sequences.

Drawbacks and Improvements Aliasing with small movements Aliasing with small movements Sensing of eye movement vs. head movement during data gathering Sensing of eye movement vs. head movement during data gathering Future Enhancements Future Enhancements –Eye/Eyelid data –More model gaze patterns –More subjects for data –Scan-path model for close-up images

Developments J. Badler, Director, Center for Human Modeling and Simulation J. Badler, Director, Center for Human Modeling and Simulation –Digital Human Modeling/Behavior –“Jack” Software –Simulation of workflow using virtual people

References Badler, Jeremy B., Badler, Norman I, Lee, Sooha Park, “Eyes Alive Badler, Jeremy B., Badler, Norman I, Lee, Sooha Park, “Eyes Alive