Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute Funded by AFOSR and Honda

Visual Behaviors Visual behaviors that typically reflect a person's level of fatigue include –Eyelid movement –Head movement –Gaze –Facial expressions

Eye Detection and Tracking

Eye Detection

Eye Tracking Develop an eye tracking technique based on combining mean-shift and Kalman filtering tracking. It can robustly track eyes under different face orientations, illuminations, and large head movements.

Eyelid Movements Characterization Eyelid movement parameters Percentage of Eye Closure (PERCLOS) Average Eye Closure/Open Speed (AECS)

Gaze (Pupil Movements) Real time gaze tracking Develop a real time gaze tracking technqiue. No calibration is needed and allows natural head movements !.

Gaze Estimation Gaze is determined by Pupil location (local gaze) Local gaze is characterized by relative positions between glint and pupil. Head orientation (global gaze) Head orientation is estimated by pupil shape, pupil position, pupil orientation, and pupil size.

Gaze Parameters Gaze spatial distribution over time PERSAC-percentage of saccade eye movement over time

Gaze distribution over time while alert

Gaze distribution over time while fatigue

Gaze distribution over time for inattentive driving

Plot of PERSAC parameter over 30 seconds.

Head Movement Real time head pose tracking Perform 3D face pose estimation from a single uncalibrated camera. Head movement parameters Head tilt frequency over time (TiltFreq)

The flowchart of face pose tracking

Examples Face Model Acquisition

Head pitches (tilts) monitoring over time (seconds)

Facial Expressions Tracking facial features Recognize certain facial expressions related to fatigue like yawning and compute its frequency (YawnFreq) Building a database of fatigue expressions for training

The plot of the openness of the mouth over time

Facial expression demo

Fatigue Modeling Observations of fatigue is uncertain, incomplete, dynamic, and from different from perspectives Fatigue represents the affective state of an individual, is not observable, and can only be inferred.

Overview of Our Approach Propose a probabilistic framework based on the Dynamic Bayesian Networks (DBN) to systematically represent and integrate various sources of information related to fatigue over time. infer and predict fatigue from the available observations and the relevant contextual information.

Bayesian Networks Construction A DBN model consists of target hypothesis variables (hidden nodes) and information variables (information nodes). Fatigue is the target hypothesis variable that we intend to infer. Other contextual factors and visual cues are the information nodes.

Causes for Fatigue Major factors to cause fatigue include: Sleep quality. Circadian rhythm (time of day). Physical conditions. Working environment.

Bayesian Fatigue Model

Dynamic Fatigue Modeling

Bayesian Fatigue Model Demo

Interface with Vision Module An interface has been developed to connect the output of the computer vision system with the information fusion engine. The interface instantiates the evidences of the fatigue network, which then performs fatigue inference and displays the fatigue index in real time.

Conclusions Developed non-intrusive real-time computer vision techniques to extract multiple fatigue parameters related to eyelid movements, gaze, head movement, and facial expressions. Develop a probabilistic framework based on the Dynamic Bayesian networks to model and integrate contextual and visual cues information for fatigue detection over time.

Effective Fatigue Monitoring The technology must be non-intrusive and in real time. It should simultaneously extract multiple parameters and systematically combine them over time in order to obtain a robust and consistent fatigue characterization. A fatigue model is needed that can represent uncertain and dynamic knowledge associated with fatigue and integrate them over time to infer and predict human fatigue.