1. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw1 A System to Detect Complex Motion of Nearby Vehicles on Freeways C. Y. Fang Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. P. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. C. S. Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C. S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C.

2. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw2 Outline Introduction Dynamic visual model (DVM) Detect complex motion of nearby vehicles Feature extraction Attention map partition Collection of classification results Temporal integral process Experimental Results Conclusions

3. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw3 Introduction -- VDAS System to detect motion of nearby vehicles: a Vision- based driver assistance system (VDAS) Advantages: High resolution Rich information Difficulties of VDAS Weather and illumination Daytime and nighttime Vehicle motion and camera vibration

4. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw4 Introduction -- DVM DVM: dynamic visual model A computational model for visual analysis using video sequence as input data Two ways to develop a visual model Biological principles Engineering principles Artificial neural networks

5. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw5 Improved DVM Two problems: The motions of vehicles may occur anywhere on the road. Training a CART neural network to recognize various complex motions is quite difficult. Solutions: Feature extraction Attention map partition Collection of classification results Temporal integral process

6. The Improved Dynamic Visual Model Conceptual component Perceptual component Category Sensory component Information acquisition CART STA neural module Yes No Video images Focuses of attention Spatialtemporal information Category Data transduction Action Episodic Memory Windowing Feature extraction Features CART Decision making Confirm? No Yes

7. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw7 Introduction Motions of the Vehicles Lane change Speed change Objective Simple motion detection Complex motion detection

8. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw8 Simple Motion Patterns

9. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw9 Attention Map Partition b4b4 b5b5 b1b1 b2b2 b3b3

10. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw10 Feature Extraction (1) --- Skewness features gi1gi1 i gi1gi1 i 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10

11. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw11 Feature Extraction (2) The horizontal skewness features: g i1 : the skewness of intensity value m i2, m i3 : the normalized second and third moments, respectively :the column means of intensity values, : the mean horizontal position of the intensity means

12. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw12 Classification Results CART 1 CART 2 CART n CART i D t

13. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw13 Temporal Fuzzy Integral (1) Let n be the number of CART neural networks : the output strings of labels of CART i from time t-r i +1 to t, k = 1, 2, …, r i : the set of all labels, where l 0 is null label p j : the stored pattern corresponding to label l j P : the set containing all stored patterns r i : time period

14. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw14 Temporal Fuzzy Integral (2) Fuzzy measure function where #p : the number of non-zero pixels of one stored pattern : the number of such pixels falling in the union of windows i or j.

15. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw15 Some Values of Fuzzy Measure Function CARTs Motion cases CART 1 CART 2 CART 3 CART 4 CART 5 (a) Vehicle ahead slows down. 0.7760.6590.7250.2900.316 (b) Right front vehicle changes lane to the left. 0.3720.0690.8630.0420.539 (c) Left front vehicle changes lane to the right. 0.3720.8630.0690.5390.042 (d) Vehicle ahead changes lane to the right. 0.3300.1240.6790.0430.335 (e) Vehicle ahead changes lane to the left. 0.3300.6790.1240.3350.043

16. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw16 Temporal Fuzzy Integral (3) Confidence function where j, k = 1, 2, …, r i, : a distance between p j and p k, : weight functions, : positive parameters

17. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw17 Temporal Fuzzy Integral (4) Fuzzy integral : the integral value for : the fuzzy intersection characterized by a t-norm

18. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw18 Intermediate Decision of Individual CART i where : a distance threshold

19. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw19 Collection of Classification Results The final classification set where, : the corresponding integral value of : a threshold

20. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw20 Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

21. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw21 Experimental Results b4b4 b5b5 b1b1 b2b2 b3b3

22. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw22 Experimental Results (1) b4b4 b5b5 b1b1 b2b2 b3b3

23. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw23 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (1)

24. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw24 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (2)

25. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw25 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (3)

26. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw26 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (4)

27. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw27 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (5)

28. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw28 b4b4 b5b5 b1b1 b2b2 b3b3 Experimental Results (6)

29. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw29 Complex Motion Sequence A B C

30. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw30 Experimental Results Simple motion sequences 12 sequences accuracy rate: 97.9% Complex motion sequences 18 sequences accuracy rate: 93.3% Since our system only outputs a result for each input sequence, this ratio is enough for our system to recognize road signs correctly.

31. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw31 Experimental Results

32. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw32 Discussion Improve attention map partition Detect other dynamic obstacles

33. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw33 Conclusions A neural-based dynamic visual model Three major components: sensory, perceptual and conceptual component Future Researches Potential applications Improvement of the DVM structure DVM implementation

34. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw34 Human Visual Process Transducer Sensory analyzer Class of input stimuli Perceptual analyzer Conceptual analyzer Physical stimuli Data compression Low-level feature extraction High-level feature extraction Classification and recognition

35. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw35 Neural Modules Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module

36. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw36 STA Neural Network (1) akak Output layer (Attention layer) njnj Inhibitory connection Excitatory connection Input layer w ij aiai xjxj nknk nini

37. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw37 STA Neural Network (2) The input to attention neuron n i due to input stimuli x : The linking strengths between the input and the attention layers corresponding neurons w kj nini njnj nknk Input neuron Attention layer rkrk Gaussian function G

38. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw38 STA Neural Network (3) The input to attention neuron n i due to lateral interaction: Lateral distance “Mexican-hat” function of lateral interaction Interaction +

39. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw39 STA Neural Network (4) The net input to attention neuron n i : : a threshold to limit the effects of noise where 1< d <0

40. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw40 STA Neural Network (5) t p 1 pd 1 The activation of an attention neuron in response to a stimulus. stimulus activation

41. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw41 ART2 Neural Network (1) CART r p u w v x q y Input vector i Input representation field F 1 Attentional subsystem Orienting subsystem G G G G G Category representation field F 2 Reset signal ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ ＋ － － － － － Signal generator S

42. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw42 ART2 Neural Network (2) The activities on each of the six sublayers on F 1 : where I is an input pattern where where the J th node on F 2 is the winner

43. National Taiwan Normal Universityviolet@ice.ntnu.edu.tw43 ART2 Neural Network (3) Initial weights: Top-down weights: Bottom-up weights: Parameters: