SAVE-IT SAfety VEhicles using adaptive Interface Technology Phase 1 Research Program Quarterly Program Review Task 5: Cognitive Distraction John D. Lee University of Iowa September 12, 2015

SAVE-IT Task Description u Team Members –John Lee (lead) –Daniel McGehee –Michelle Reyes –Tara Smyser –Yulan Liang u Objectives –Develop an equation relating driving performance, driver state, and vehicle system state variables associated with cognitive distraction to reaction time u Deliverables –A report based on the literature review, the identification of potential measures of cognitive distraction, and an updated task definition document (for Task 5A) –A final report that describes the design, research methodology, data analysis, resultant data, and recommendations, including diagnostic measures and preliminary algorithms expressing the relationship between RTC and oculometric and performance measures (for Task 5B)

SAVE-IT Task Description u Schedule Task 5.A Literature review –5.A.1 Review the relationship between eye movements, ocular response, scan patterns, and other psycho-physiological measures and cognitive load (complete) –5.A.2 Review range of driving tasks and develop a theory-based taxonomy of different subtypes of cognitive distraction (complete) –5.A.3 Identify neuropsychological tests that tap into the various cognitive demands identified in Task 5.A.2 (complete) –5.A.4 Develop a conceptual, and potentially, a computational model of driver response to the cognitive demands of driving and in-vehicle device interactions (in progress)

SAVE-IT Task description u Schedule Task 5.B Data collection and algorithm development –5.B.1 Identify a small set of neuropsychological tasks for implementation in a driving scenario (in progress) –5.B.2 Develop scenarios and driving performance measures (in progress) –5.B.3 Collect data and evaluate the relationship between driver performance and behavior and degree of cognitive distraction (in progress)

SAVE-IT Literature Review: Task 5A u Research Areas of Interest –Empirical and theoretical descriptions of driver distraction –Theoretical basis for multitask performance and task management –Measures that might predict distraction –Algorithms for combining data to predict distraction u Key Source Material –Boer, E. R. (2001). Behavioral entropy as a measure of driving performance. Proceedings of the First Biannual Driving Assessment Conference, Aspen, CO. –Hart, S. G. and C. D. Wickens (1990). Workload assessment and prediction. MANPRINT: An emerging technology. Advanced concepts for integrating people, machines and organizations. H. R. Booher. New York, Van Nostrand Reinhold: –Lee, J. D. and T. F. Sanquist (2000). "Augmenting the operator function model with cognitive operations: Assessing the cognitive demands of technological innovation in ship navigation." IEEE Transactions on Systems, Man, and Cybernetics- Part A: Systems and Humans 30(3): –Tijerina, L., S. Kiger, et al. (1995). Final report - Workload assessment of in-cab text message system and cellular phone use by heavy vehicle drivers on the road. (Contract No. DTNH ). DOT HS (7A). Washington, DC. –Wickens, C. D. (2002). "Multiple resources and performance prediction." Theoretical Issues in Ergonomics Science 3(2): –Wierwille, W. W. (1993). Visual and manual demands of in-car controls and displays. Automotive Ergonomics. B. Peacock and W. Karwowski. Washington, DC, Taylor and Francis: –Young, M. S. and N. A. Stanton (2002). "Malleable attentional resources theory: A new explanation for the effects of mental underload on performance." Human Factors 44(3):

SAVE-IT Literature Review: Task 5A u Major findings: General approach –Convergent estimate of distraction induced reaction time decrements »Driver state »Driver performance »IVIS and roadway state and associated cognitive demand –Models augment a data-driven prediction of distraction »Multiple resource theory »OFM-COG –Potential algorithms to relate variables to distraction estimate »Factor analysis and linear regression »Hidden Markov model »Vector Support Machines –Theory-based considerations of distraction »Multiple resource overload »Malleable resources—underload and workload transitions »Strategic task management, decision to engage and switching efficiency »Multiple levels of behavior (control, strategic, and tactical)

SAVE-IT Literature Review: Task 5A ● = Very appropriate ⊙ = Somewhat appropriate ○ = Inappropriate MeasureTimelyDiagnosticSensitivePractical Eye movements and scan patterns ●●● ⊙ Eye blinks ● ⊙⊙⊙ Pupil dilation ● ⊙ ○ ⊙ Cardiac measure—0.1 Hz variability ⊙ ●● ⊙ Cardiac measure—Vagal tone ●●●○ Respiration rate ● ⊙ ○○ Blood pressure ⊙ ●●○ Electrodermal activity ○ ⊙ ○○ Electroencelphalogram (EEG) ●●●○ Event related potential (ERP) ●●●○ Hormone levels ○ ⊙ ○○ Facial electromyography (EMG) ● ⊙⊙ ○

SAVE-IT Literature Review: Task 5A MeasureTimelyDiagnosticSensitivePractical Driving performance—Lane position ⊙⊙⊙⊙ Driving performance—Speed control ⊙ ○ ⊙ ● Driving performance—Headway maintenance ⊙⊙ ●● IVIS interactions—MRT-based estimation ● ⊙⊙⊙ IVIS interactions—ACT_R-based estimation ● ⊙⊙⊙ IVIS interactions—OFM-COG-based estimation ● ⊙⊙⊙ ● = Very appropriate ⊙ = Somewhat appropriate ○ = Inappropriate

SAVE-IT Multiple resources describing cognitive demands

SAVE-IT Cognitive transactions from OFM-COG Cognitive Task General Category of Information Processing Human Information Processing Resources 2.Filter. Straining out what does not matter.AcquisitionSelective attention 4.Search. Looking for something.AcquisitionSustained attention, Perceptual sensitivity 19.Decide/Select. Choosing a response to fit the situation. InterpretLong-term memory, Processing strategy 20.Plan. Matching resources in time to expectations. InterpretWorking memory, Processing strategy 21.Test. Is it what is should be?InterpretPerceptual sensitivity, Working memory, Long-term memory 22.Interpret. What does it mean?InterpretationLong-term memory, Sustained attention 23.Categorize. Defining and naming a group of things. InterpretationLong-term memory, Perceptual sensitivity

SAVE-IT Literature Review: Task 5A u Major Findings: Algorithms to predict distraction –Hidden Markov Models (HMMs) »Represent the stochastic sequences as Markov chains where the states are not directly observed but are associated with a probability density function »HMMs are used to build a signal model that explains and characterizes the occurrence of observed signal outputs. These models can then be used at a later time to identify or recognize the signal output patterns of other sequences. –Support Vector Machines (SVMs) »The objective is to find an optimal hyperplane which correctly classifies as many data points as possible and to separate the points of two classes as far as possible. It is a constrained OPTIMIZATION problem. »SVMs map training data into the high-dimensional feature space, and construct a separating hyperplane with maximum margin. This yields a nonlinear decision boundary in input space.

SAVE-IT Literature Review: Task 5A The red dots indicate distraction and the blue dots indicates no distraction. u SVM is useful to classify the different patterns of eye movements which indicate potential periods of distraction u 15-minute period eye movement data are used as the training data. Each training point is presented as a 2-dimension vector: (fixation frequency, standard deviation of saccade length) in 3 seconds prior.

SAVE-IT Literature Review: Task 5A u Summary/Hypotheses –Driver-specific models may be more effective than generic models u Multiple types of distraction are important –Information overload and task conflict as described by multiple resource demand vectors and conflict matrices –Malleable resource theory predicts low workload can diminish resources and impair performance during transitions to high workload periods –Task preemption and attentional withdrawal –Multi-task dynamics »Task inertia leads to switching costs »Poor strategic task management—decision to engage during high demand periods –Failures at the control and tactical levels of driving performance

SAVE-IT Literature Review: Task 5A u Summary/Hypotheses –Distraction is best predicted by estimating task demands on the driver, driver state, and driver performance –IVIS and roadway demands can be characterized by »an eight-value vector that describes the type and amount of resources demanded »a conflict matrix that identifies the degree to which common use of each of the eight resources will degrade performance »OFM-COG provides a useful structure to capture higher level aspects of task complexity –Driver state can be characterized by: »Eye movements »Eye blinks » 0.1 Hz variability of heart rate –Driving performance can be characterized by time based measures of »Lane position (time to lane crossing) »Speed control »Headway maintenance (Time to contact and time headway) –Interactions between variables may be particularly good predictors of distraction (e.g., breakdowns in scanning patterns during passing or curve approaches)

SAVE-IT Research: Task 5 B u Objectives –Assess how well IVIS interactions, physiological, and driving performance measures, alone and in combination, can predict reaction time decrements –Investigate whether distraction affecting tactical performance displays a different signature compared to distraction affecting control performance –Assess how individual and generic models affect the precision of the predicted distraction –Assess the robustness, reliability, and specificity of traditional linear regression, Hidden Markov Models, and Support Vector Machines in predicting driver distraction

SAVE-IT Research: Task 5 B u Independent variables: –OFM-COG task type and MRT task demand (24 IVIS tasks total) –Driving events (control response, tactical response) u Dependent variables –Eye blink, Eye fixation and gaze history, Heart rate –Steering, speed, and headway control –Reaction time to safety-relevant driving events u Apparatus/Subjects –24 participants between years of age u Design –Fully within subjects design (IVIS task X Driving Event) u Issues/Concerns (include potential impact on schedule and solutions) –Accuracy and integration issues with Seeing Machines system

SAVE-IT Research: Task 5 B u Expected Results –Equation predicting distraction-related decrements in driver reaction time –Assessment of the viability of three-component convergent approach to estimating driver distraction –Assessment of prediction algorithms (Linear regression, HMM, and SVM) –Different tasks degrade tactical and control performance –Different predictors for degraded tactical and control performance