1. Evaluating Pre-Processing Pipelines for Thermal-Visual Smart Camera
Authors:
Irida Shallari, Muhammad Imran, Najeem Lawal, Mattias O’Nils

2. Embedded processing in smart camera
Benefits
Reduced data for communication and analysis
Real-time monitoring
Challenges
Energy consumption
Performance
Limited resources
Computational
Memory
Energy
Communication bandwidth

3. Embedded processing in smart camera

4. Surveillance of vulnerable areas

5. Alternative approach
High frequency
Low frequency

6. Problem statement
Low level
Design exploration in pixel-based image pre-processing pipeline for multi-sensor smart camera with respect to data communication vs classification accuracy.
Intermediate
High level
-Spatial filtering
-Temporal filtering
-Segmentation
-Morphology
-Labelling
-Classfication
-Recognition
Xmend trade-off communication vs performance

7. Classification algorithms
1280×1024
320×240
Capturing
Capturing
ROI
Segmented ROI
Segmented ROI
ROI
Classification algorithms
SURF, SIFT
Pre-processing architecture for smart camera

8. Image dataset
Human and cyclist

9. Experimental setup Nexys 4 board which includes Artix-7 FPGA
NVIDIA Tegra TK1
An IDS CMOS µEye visual camera
focal length 12 mm, a resolution of 1280×1024 and a pixel pitch of 5.3 μm
A Tamarisk 320 thermal camera
focal length 19 mm, a sensor resolution of 320 x 240 and a pixel pitch of 17 μm

10. Design exploration
Thermal
Thermal binary
Visual
Visual binary

11. Architecture Camera node Client device 320×240 Human Animal Cyclist
Capture
Background subtract
Camera node
Segmentation
Morphology
Binary ROI coding
Decoding
Classification
Client device
Human
Animal
Cyclist

12. Classification accuracy vs Communication cost

13. Alternative approach
High frequency
Low frequency

14. Proposed architecture

15. Classification accuracy

16. Data compression (1280×1024) 3840 (200×300) 1406 (250×500) 2930 3 4 59
Image type
Res./Size
Kbytes
(Raw data)
Kbytes (RAW_ROI
Kbytes (JPEG_ROI)
KBytes (Bin_ROI)
KBytes (G4_ROI)
Human
Cycl.
Visual
(1280×1024)
3840
(200×300)
1406
(250×500)
2930 3 4 59
122 2
Thermal
(320×240) 75
(100×150)
117
(100×250) 20 1 15

17. Conclusions We propose an architectural approach in which:
Thermal images are used as a mask to extract ROI
Frequently transmitted ROI visual data
Compressed visual data transmitted rarely for situational awareness
Visual compressed ROI offers:
13%-64% better classification accuracy than binary visual ROI or thermographic images (raw_ROI, bin_ROI)
New applications requiring situational awareness.

18. Questions?

19. Thank you!