Multi-Sensor Soft-Computing System for Driver Drowsiness Detection Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Institute of Integrated Sensor Systems Department of Electrical and Computer Engineering Multi-Sensor Soft-Computing System for Driver Drowsiness Detection Li Li, Klaudius Werber, Carlos F. Calvillo, Khac Dong Dinh, Ander Guarde and Andreas König 10-Dec-2012 Introduction Driving Scene Modeling and Hardware Setup Software Components and Algorithms Experimental Results Conclusion and Future Work Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Introduction Major factor in 20 percent of all accidents in the United States in 2006 The second most frequent cause of serious truck accidents on German highways Major damage caused by drowsy truck or bus drivers Enhance active safety with advanced driver assistance Andreas König, 2001

Hardware Setup DeCaDrive System Multi-sensing interfaces Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Hardware Setup DeCaDrive System Multi-sensing interfaces Depth camera Steering angle sensor Pulse rate sensor … … PC-based soft-computing subsystem PC-based driving simulator Andreas König, 2001

Hardware Setup SoA depth camera Extention of 2D image with distance Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Hardware Setup SoA depth camera Extention of 2D image with distance Wide field of view Relatively low computational cost Robust to lighting variations (active sensing) Non-intrusive and non-obstructive (eye-safe NIR light source) Microsoft Kinect PMD CamCube SoftKinetic DepthSense Make them suitable for driver status monitoring in various lighting conditions, especially in truck driver cabin. Andreas König, 2001

Hardware Setup Pulse rate sensor Steering angle sensor Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Hardware Setup Pulse rate sensor Heart health and fitness Time domain analysis Frequency domain analysis Steering angle sensor Steering behavior of driver Correlation with driver status and driver intention embedded Andreas König, 2001

Software Components and Algorithms Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Software Components and Algorithms Overview of the data processing flow Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Feature Computation Features being computed from multiple sensor measurements Andreas König, 2001

Experimental Results Test subjects Experiments Five male test subjects Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Test subjects Five male test subjects 22 to 25 years old (mean: 23.6, std:1.1) All have driver‘s license for at least 4 years No alcohol drinking before test Experiments One hour driving simulation for each test subject 588-minute driving sequence recorded Ground truth: not drowsy, a little drowsy, deep drowsy Through self-rated score and response time Andreas König, 2001

Experimental Results Examples of different sensor features Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Examples of different sensor features blink frequency low steering percentage mean pulse rate Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Screenshot of online processing of various sensor data Eye pupil and corners Depth image Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Results of ANN based classifier with two training algorithms Andreas König, 2001

Experimental Results Confusion matrix of SCG Confusion matrix of LM Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Confusion matrix of SCG 40 hidden neurons 10-fold cross-validation Confusion matrix of LM 80 hidden neurons 10-fold cross-validation Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Experimental Results Drowsiness level classification accuracy depending on selected features Andreas König, 2001

Conclusion and Future Work Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Conclusion and Future Work Contribution Emerging framework for driver status monitoring and intention detection with multi-sensor soft-computing system Classification of three different drowsiness levels with up to 98.9% accuracy based on data sets of five test subjects. Future work Validation with more statistics and with data from real vehicles Variance compensation by adaptive learning Optimization of feature selection with sophisticated heuristics Utilization of other advanced classification techniques, e.g., SVM Integration of more embedded sensors with wireless technology In this paper, we present a novel system approach to driver drowsiness detection based on multi-sensor data fusion and soft-computing algorithms. By observing head movement and facial features with depth camera, by monitoring steering behavior and pulse rate of driver the presented system is able to classify three different drowsiness levels with up to 98.9% accuracy based on data sets of five test subjects. The robustness of the presented approach needs to be validated with more statistics and with data from real vehicles. The variance of drivers, vehicles, road and weather conditions can be compensated by adaptive learning. Feature selection can be optimized with sophisticated heuristics, e.g., genetic algorithm (GA) and particle swarm optimization (PSO). Advanced classification techniques such as support vector machine (SVM) can be incorporated in the system as well for further optimization. In addition, we plan to integrate more miniaturized embedded sensors with wireless technology, e.g., EEG and ECG sensors, to improve the effectiveness and robustness of the system. Andreas König, 2001

Computer-Aided Design of Analog and Mixed-Signal ICs using Cadence DFW II icfb Thank you! Andreas König, 2001