1. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Context Enhancement of Nighttime Surveillance by Image Fusion Yinghao Cai Kaiqi Huang, Tieniu Tan and Yunhong Wang Center for Biometrics Research and Testing National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences 2006-8-21

2. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Outline Motivation Proposed Method Conclusions

3. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences (a) Nighttime Image (b) Daytime background (c) Result of context enhancement

4. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Motivation Few work has been done on nighttime surveillance. Difficulties: Low contrast Low signal to noise ratio Limited environmental information( context information)

5. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Solution The camera is fixed. Capture scenes of day and night at the same viewpoint. Make use of the high quality background of the day to help enhance the context of nighttime images.

6. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Questions What information should be preserved in the fused image?  Moving objects  Illumination effects  Daytime background Moving objects and illumination effects preserve the fidelity of the important information of the nighttime video

7. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Framework Nighttime image Enhanced image Nighttime image Illumination segmentation Final image Daytime background

8. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Enhancement of nighttime video A tone mapping function is employed to nighttime video enhancement [Bennett, 2005]. Where is the pixel of the original nighttime video, is the value of the enhanced video, is a parameter.

9. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Comparison with Gamma Correction (a) Original video (b) By [Bennett, 2005] ’s method(c) By Gamma correction

10. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Motion Detection Gaussian mixture models [Stauffer, 99]. Real time motion detection. Robust to variations in lighting, moving scene clutter, multiple moving objects.

11. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences (a) Original nighttime video (b) Enhanced video (c) Motion detection of original video(d) Motion detection of enhanced video C. Stauffer and W.E.L.Grimson, adaptive background mixture models for real-time tracking, CVPR 99

12. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Estimation of illumination characteristics = The image I can be represented by the product of reflectance of the scene R, and illumination coming from the light source L. In Retinex theory, the illumination can be considered as the low frequency of image. In this paper, we represent the illumination characteristics of the nighttime image as the smoothed version of the original image. Image Reflectance Illumination

13. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Image fusion Where F is the final image, N is the nighttime image, D is the daytime background. W is the weight: Where M is the result of motion detection, L is illumination characteristic. They are both in the range [0,1].

14. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Experimental Results (a) Original nighttime image (b) Result of context enhancement

15. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Experimental Results (a) Original nighttime video(b) Result of context enhancement

16. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Comparison with related work [Li, 2005]’s method Our method

17. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences Conclusions Simple but effective. Provides a real time and robust solution to front- end image pre-processing in nighttime surveillance. The resultant image contains a more accurate and comprehensive description of the scene which is more useful for human visual and machine perception, especially in surveillance.

18. 模式识别国家重点实验室 中国科学院自动化研究所 National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences yhcai@nlpr.ia.ac.cn