NTU CSIE 研 究 生：方瓊瑤 A Vision-Based Driver Assistance System Based on Dynamic Visual Model 指導教授：陳世旺博士 傅楸善博士

NTU CSIE Outline Introduction Dynamic visual model (DVM) Neural modules Road sign recognition system System to detect changes in driving environments System to detect motion of nearby moving vehicles Conclusions

NTU CSIE Introduction (1) -- ITS Intelligent transportation system (ITS) Advanced traffic management systems (ATMS) Advanced traveler information systems (ATIS) Commercial vehicle operations (CVO) Advanced public transportation systems (APTS) Advanced rural transportation systems (ARTS) Advanced vehicle control and safety systems (AVCSS) Driver assistance systems (DAS)

NTU CSIE Introduction (2) -- DAS Driver assistance systems (DAS) Safety Passive Active Driving is a sophisticated process The technology of vehicle The temperament of the driver

NTU CSIE Introduction (3) -- VDAS Vision-based driver assistance systems (VDAS) Difficulties of VDAS Weather and illumination Daytime and nighttime Vehicle motion and camera vibration

NTU CSIE Introduction (4) Subsystems of VDAS Road sign recognition system System to detect changes in driving environments System to detect motion of nearby moving vehicles

NTU CSIE Introduction (5) -- DVM DVM: dynamic visual model A computational model for visual analysis using video sequence as input data Two ways to develop a visual model Biological principles Engineering principles Artificial neural networks

NTU CSIE Dynamic Visual Model Conceptual component Perceptual component Sensory component Information acquisition CART neural module STA neural module Yes No Video images Focuses of attention Spatialtemporal information Categorical features Category Feature detection Pattern extraction CHAM neural module Patterns Data transduction Action Episodic Memory

NTU CSIE Human Visual Process Transducer Sensory analyzer Class of input stimuli Perceptual analyzer Conceptual analyzer Physical stimuli Data compression Low-level feature extraction High-level feature extraction Classification and recognition

NTU CSIE Neural Modules Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module Configurable heteroassociative memory (CHAM) neural module

NTU CSIE STA Neural Network (1) akak Output layer (Attention layer) njnj Inhibitory connection Excitatory connection Input layer w ij aiai xjxj nknk nini

NTU CSIE STA Neural Network (2) The input to attention neuron n i due to input stimuli x : The linking strengths between the input and the attention layers corresponding neurons w kj nini njnj nknk Input neuron Attention layer rkrk Gaussian function G

NTU CSIE STA Neural Network (3) The input to attention neuron n i due to lateral interaction: Lateral distance “Mexican-hat” function of lateral interaction Interaction +

NTU CSIE STA Neural Network (4) The net input to attention neuron n i : : a threshold to limit the effects of noise where 1< d <0

NTU CSIE STA Neural Network (5) t p 1 pd 1 The activation of an attention neuron in response to a stimulus. stimulus activation

NTU CSIE ART2 Neural Network (1) CART r p u w v x q y Input vector i Input representation field F 1 Attentional subsystem Orienting subsystem G G G G G Category representation field F 2 Reset signal ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ － － － － － Signal generator S

NTU CSIE ART2 Neural Network (2) The activities on each of the six sublayers on F 1 : where I is an input pattern where where the J th node on F 2 is the winner

NTU CSIE ART2 Neural Network (3) Initial weights: Top-down weights: Bottom-up weights: Parameters:

NTU CSIE HAM Neural Network (1) CHAM j Output layer (Competitive layer) Excitatory connection Input layer w ij xjxj i vivi v1v1 v2v2 vnvn

NTU CSIE HAM Neural Network (2) The input to neuron n i due to input stimuli x : n c : the winner after the competition

NTU CSIE Road Sign Recognition System

NTU CSIE Objective Get information about road Warn drivers Enhance traffic safety Support other subsystems

NTU CSIE Problems

NTU CSIE Perceptual Component

NTU CSIE Conceptual Component— C lassification results of CART Training Set Test Set

NTU CSIE Conceptual Component — Training Patterns for CHAM

NTU CSIE Experimental Results

NTU CSIE

NTU CSIE

NTU CSIE Other Examples

NTU CSIE Discussion Vehicle and camcorder vibration Incorrect recognitions Input patterns Recognition results Correct patterns

NTU CSIE System to Detect Changes in Driving Environments

NTU CSIE Definition The environmental changes in expressways: Left-lane-change Right-lane-change Tunnel-entry Tunnel-exit Expressway-entry Expressway-exit Overpass-ahead

NTU CSIE Objective Coordinate DAS subsystems Update parameters Detect unexpected changes Detect rapid changes

NTU CSIE Results of the Sensory Component

NTU CSIE Results of the Perceptual Component

NTU CSIE The Prototypical Attention Patterns

NTU CSIE Experimental Results

NTU CSIE Experimental Results

NTU CSIE Experimental Results

NTU CSIE Experimental Results

NTU CSIE Experimental Results

NTU CSIE Discussion Curved roads Shadows Multiple environmental changes

NTU CSIE System to Detect Motion of Nearby Moving Vehicles

NTU CSIE Introduction Motions of the Vehicles Lane change Speed change Objective Simple motion detection Complex motion detection

NTU CSIE Simple Motion Patterns

NTU CSIE Improved DVM Two problems: The motions of vehicles may occur anywhere on the road. Training a CART neural network to recognize various complex motions is quite difficult. Solutions: Feature extraction Attention map partition Collection of classification results Temporal integral process

NTU CSIE No Yes Attention maps Windowing Feature extraction CART 1 CART 2 CART n-1 CART n Decision making Confirm? Output bnbn b n-1 b2b2 b1b1 st1st1 st2st2 s t n-1 stnstn Feature extraction Flowchart for Conceptual Component

NTU CSIE Attention Map Partition b4b4 b5b5 b1b1 b2b2 b3b3

NTU CSIE Feature Extraction (1) --- Skewness features gi1gi1 i gi1gi1 i

NTU CSIE Feature Extraction (2) The horizontal skewness features: g i1 : the skewness of intensity value m i2, m i3 : the normalized second and third moments, respectively :the column means of intensity values, : the mean horizontal position of the intensity means

NTU CSIE Classification Results CART 1 CART 2 CART n CART i D t

NTU CSIE Temporal Fuzzy Integral (1) Let n be the number of CART neural networks : the output strings of labels of CART i from time t-r i +1 to t, k = 1, 2, …, r i : the set of all labels, where l 0 is null label p j : the stored pattern corresponding to label l j P : the set containing all stored patterns r i : time period

NTU CSIE Temporal Fuzzy Integral (2) Fuzzy measure function where #p : the number of non-zero pixels of one stored pattern : the number of such pixels falling in the union of windows i or j.

NTU CSIE Some Values of Fuzzy Measure Function

NTU CSIE Temporal Fuzzy Integral (3) Confidence function where j, k = 1, 2, …, r i, : a distance between p j and p k, : weight functions, : positive parameters

NTU CSIE Temporal Fuzzy Integral (4) Fuzzy integral : the integral value for : the fuzzy intersection characterized by a t-norm

NTU CSIE Intermediate Decision of Individual CART i where : a distance threshold

NTU CSIE Collection of Classification Results The final classification set where, : the corresponding integral value of : a threshold

NTU CSIE Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

NTU CSIE Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

NTU CSIE Complex Motion Sequence A B C

NTU CSIE Experimental Results Simple motion sequences 12 sequences accuracy rate: 97.9% Complex motion sequences 18 sequences accuracy rate: 93.3%

NTU CSIE Experimental Results

NTU CSIE Discussion Improve attention map partition Detect other dynamic obstacles

NTU CSIE Conclusions A neural-based dynamic visual model Three major components: Sensory component Perceptual component Conceptual component Three DAS subsystems: Road sign recognition system Driving environmental change detection system Nearby vehicle motion detection system

NTU CSIE Future Research Potential applications Lane marks detection Obstacle recognition Drowsy driver detection Coordinating system Improvement of the DVM structure Acoustic and tactile processes Reasoning: Intellectual and associative thinking DVM implementation Genetic algorithms Fuzzy theoretic methods