Image Processing IP cores © eVS Srl, May 30, 2005

Vision, Image Processing and Sound Lab Cooperation University of Verona Department of Computer Science Ultimodule Vision, Image Processing and Sound Lab Confidential © 2005 eVS Srl.

Research Areas Image Processing Computer Vision Pattern Recognition Image Restoration Feature Extraction (Edge, Corner, Region, Gradient) Computer Vision Mosaics and Video summarization Model-Based Tracking Stereopsis and 3D reconstruction Pattern Recognition Statistical techniques Clustering and classifications of sequences Object Recognition Confidential © 2005 eVS Srl.

Applications Security and Monitoring Industrial Marine Automotive Video surveillance Intrusion detection Industrial Active video surveillance Robotic arm motion Measurement of points and distances Marine Auto docking Support to navigation (stabilization) Hazard detection Automotive Active security Autonomous guided vehicles Confidential © 2005 eVS Srl.

Sensing The Real World Stereo sensor to reconstruct the 3rd dimension Robustness to brightness changes on scene Real time performance Output Image robust RGBZ resolution up to 512x512 up to 100 fps Confidential © 2005 eVS Srl.

Image Processing Engine The image processing engine implements most of the algorithms required when an application is asked to rely on vision to make decisions and control Real time performance High computational power Self-contained modules Compact configuration Common interface High modularity for different contexts Optical (2D) and range (3D) image support Confidential © 2005 eVS Srl.

Stereopsis 1 The computational stereopsis is the process used to obtain the depth information from a pair of images coming from two cameras that see the same scene from two different points of view. We can distinguish two sub-problems: Corresponding points detection consists in finding which points in the left and right images are projection of the same point of the scene. The images must be rectified in order to ensure that corresponding points belong to the same horizontal line. Rectification reduces the algorithm complexity. 3D reconstruction: once the correspondences, the relative position of cameras, and the internal sensor parameters are known, the 3D position in the scene can be calculated for each point of the two 2D images. Confidential © 2005 eVS Srl.

Stereopsis 2 Left view Right view Disparity Disparity hot = close brigh = close Confidential © 2005 eVS Srl.

Left image rectification Right image rectification RGBZ sensor Stereo Post processing Stereo matching Left image rectification Z reconstruction RGBZ image Right image rectification Confidential © 2005 eVS Srl.

Image Processing Engine: a possible configuration Project Overview Image Processing Engine: a possible configuration Post- processing Corner RGBZ Sensor Motion estimation Filtering Gradient Edge Warping Segmentation Confidential © 2005 eVS Srl.

Filtering The noise is reduced by convolving the intensity image with a gaussian kernel (linear filtering) The amount of filtering can be controlled by changing the coefficients of the convolution mask Filtering Noisy image (Gaussian noise σ = 0.01) Filtered image (5x5 Gaussian filter σ = 1) Confidential © 2005 eVS Srl.

Segmentation Segmentation is usually one of the first steps in image analysis The purpose of image segmentation is to subdivide an image into meaningful, non-overlapping regions Our implementation produces as a result a binary image which distinguishes between background and foreground threshold Input image Histogram Segmented image Confidential © 2005 eVS Srl.

Edge detection An edge is a set of connected pixels that lie on the boundary between two regions To detect an edge we apply a threshold to the magnitude of image gradient Gradient is computed by convolving the image with a 3x3 Prewitt mask, and edges are thinned applying non-maximum suppression Edge Detection Original image Edge Confidential © 2005 eVS Srl.

Corner detection Corner points are detected by a significant change of the gradient values along two directions The core receives in input the image gradients and the required number of corners, and outputs the coordinates and the degree of confidence of the extracted corners. Corners are easy to track Once the position of corners is known along the video sequence, many informations on camera motion can be retrived. Confidential © 2005 eVS Srl.

Warping Warping consists in applying a geometrical spatial transformation T:R2→R2 to the image coordinates, and re-sampling the grid Some features of our image warping module: affine transformation backward mapping bilinear interpolation Confidential © 2005 eVS Srl.

Summary 1 SPACE Slices BRAM DSP48s Filtering 184 2 3 Edge-detection 1195 5 Corner-detection 1799 14 8 Warping 422 4 Segmentation 858 Rectification 1312 Stereo n.a. Performances estimation on Xilinx Virtex4 LX60 with 8 bit depth 512x512 input images Confidential © 2005 eVS Srl.

Summary 2 TIME Max Freq (MHz) Pixel Rate (MHz) Frame rate (fps) Filtering 197 49 200 Edge-detection 53 9 36 Corner-detection 42 21 86 Warping 10 26 105 Segmentation 91 370 Rectification 209 52 212 Stereo n.a. Performances estimation on Xilinx Virtex4 LX60 with 8 bit depth 512x512 input images Confidential © 2005 eVS Srl.

Work in progress Motion estimation: frame to frame geometric image transformation computation Video stabilization: unwanted motion compensation 2D Feature Tracker: correlation-based tracker of a region of interest Hough Transform: straight lines detection Confidential © 2005 eVS Srl.

Development tools Matlab: Xilinx System Generator for DSP 7.1: Algorithm design Prototyping High level verification Xilinx System Generator for DSP 7.1: Hardware design Simulation Hardware co-simulation (via JTAG) Initial resources estimation Xilinx ISE 7.1: Implementation Integration Mentor ModelSim SE 6.0: Confidential © 2005 eVS Srl.

Development boards 3 Xilinx Spartan3 Starter Kit Boards 1 ML310 200,000 gates Xilinx Spartan3 XC3S200FT256 FPGA 2Mbit Xilinx XCF02S Platform Flash 1M-byte of Fast Asynchronous SRAM 1 ML310 Virtex-II Pro XC2VP30 256MB DDR memory 512MB CompactFlash card multiple PCI slots 1 SCM40 NEC VR4131 MIPS DDR SDRAM 200,000 gates Spartan2 Confidential © 2005 eVS Srl.

Contacts eVS is: Prof. Vittorio Murino: vittorio.murino@evsys.net Dott. Roberto Marzotto: roberto.marzotto@evsys.net Dott. Alessandro Negrente: alessandro.negrente@evsys.net Dott. Marco Monguzzi: marco.monguzzi@evsys.net Phone / fax: +39 045 802 7027 Web: www.evsys.net Confidential © 2005 eVS Srl.