Introduction to IPL and OpenCV libraries Bogdan Raducanu Centre de Visiò per Computador E-mail: bogdan@cvc.uab.es

Cover Story OpenCV was of key use in the vision system of "Stanley"

What is IPL? - The IPL (Image Processing Library ) is a collection of functions implementing several image processing algorithms. It was developed by INTEL. - Is optimized for MMX and different processor types (there is a DLL for each type of INTEL processor) - The images are stored in a specific data structure. In order to work with the image, we need to know the information contained in the structure header (contains image specific characteristics)

Image structure IplImage * IplImage Data Width Height Bits per pixel Channel sequence … pointer to image data

- nChannels: number of channels The “IplImage” structure includes a header containing image information and attributes: - nChannels: number of channels (1 for grayscale, 3 for RGB, 4 for CMYK, ...) depth: number of bits/pixel and data type IPL_DEPTH_1U (1-bit) IPL_DEPTH_8U (8-bit unsigned) IPL_DEPTH_8S (8-bit signed) IPL_DEPTH_16U (16-bit unsigned) IPL_DEPTH_16S (16-bit signed) IPL_DEPTH_32S (32-bit signed) IPL_DEPTH_32F (32-bit float)

colorModel: “GRAY”, “RGB”, “CMYK”, etc. channelSeq: “GRAY”, “BGR”, “BGRA”, “RGB”, “RGBA”, “HSV”, “YUV”, etc. dataOrder: RGBRGBRGB... or RRR...GGG...BBB origin: Top-Left or Bottom-Left (IPL_ORIGIN_TL ó IPL_ORIGIN_BL) scanline alignment: DWORD or QWORD width (in pixels) height (in pixels) ROI: (could be NULL) maskROI: (could be NULL) imageSize: image size (in bytes) - imageData: pointer to pixel data

Function categories in IPL: - create/destroy an image and access its content arithmetical/logical operations filtering - morphological operations color space conversion histogram linear/geometrical transformations image statistics

Create/destroy an image and access its content: - iplCreateImageHeader iplAllocateImage iplCreateROI iplSetROI iplCopy iplClone iplDeallocateImage iplPutPixel iplGetPixel

Example (create/destroy an image): #include ”ipl.h” … IplImage *img = iplCreateImageHeader( 3, 0, IPL_DEPTH_8U, “RGB”, “BGR”, IPL_DATA_ORDER_PIXEL, IPL_ORIGIN_TL, IPL_ALIGN_QWORD, 150, 100, NULL, NULL, NULL, NULL ); iplAllocateImage(img, 0, 0); /////// Use the image //////// iplDeallocate(img, IPL_IMAGE_ALL);

We created an image of 150x100 pixels with 3 channels We created an image of 150x100 pixels with 3 channels. The color model is RGB and data type is 8 bits/pixel, unsigned. The channel order is BGRBGRBGR… starting from the upper row. The data are aligned in memory as QWORD (64 bits). There is no ROI defined We allocated memory for data, but without initialize it. With the IPL_IMAGE_ALL parameter, we freed the header of the structure, the data and the existing ROIs (if any)

- The content of an IplImage can be accessed in several ways: - using the functions GetPixel and PutPixel: Inconvenience: slow access - go directly to the memory address corresponding to the pixel al pixel: - Inconvenience: this operation is ‘complex’ because we have to compute the memory address beforehand careful with the data types - Advantage: is faster to access a big chunk of data

- Example: Let’s assume we have a RGB image with 8 bits/pixel, unsigned: #include ”ipl.h” … IplImage *img = iplCreateImageHeader( 3, 0, IPL_DEPTH_8U, “RGB”, “BGR”, IPL_DATA_ORDER_PIXEL, IPL_ORIGIN_TL, IPL_ALIGN_QWORD, 150, 100, NULL, NULL, NULL, NULL ); iplAllocateImage(img, 0, 0); unsigned char *R,*G,*B; B = (unsigned char *) img->imageData; G = B+1; R = G+1; for (int i=0; i<15000; i++, R+=3, G+=3, B+=3) { ///// here we can use/modify the pixel data } iplDeallocate(img, IPL_IMAGE_ALL);

C:\Archivos de Programa\Intel\plsuite\doc\iplman.pdf Remarks: in order to be able to use the IPL functions, you must include the header file “ipl.h” in your source code include the “ipl.lib” in the project settings the structures’ names use the “Ipl” (“I” uppercase) prefix, meanwhile the functions’ names use the “ipl” (“i” lowercase) prefix the remaining function categories will be presented in the section dedicated to OpenCV library for more information, consult the IPL manual: C:\Archivos de Programa\Intel\plsuite\doc\iplman.pdf

OpenCV library Why OpenCV? IPL is a “low-level” library (it allows basic operations) OpenCV is a library which contains more complex data structures and “high-level” functions: optical flow, pattern recognition 2D-3D real-time tracking, camera calibration, etc. it comes with some extensions that allow: accessing a camera or working with AVI files graphical user interface (“HighGUI”) allowing a faster and easier way to interact/visualize the images

OpenCV is in general compatible with the IPL library OpenCV is in general compatible with the IPL library. It is based also on the “IplImage” structure. But it must be employed taking into account the following restrictions: - the image statistics functions require that “IplImage” be defined either with a single channel or three channels of the following data types: IPL_DEPTH_8U, IPL_DEPTH_8S or IPL_DEPTH_32F. OpenCV supports only interleaved images the following attributes: colorModel, channelSeq, BorderMode, and BorderConst are ignored the attibutes maskROI and tileInfo must be set to 0. the ROIs of the input and output image must be the same. Remark: the structures’ name uses the “Cv” (“C” uppercase) prefix, meanwhile the functions’ name uses the “cv” (“c” lowercase) prefix.

Create/Destroy an image and access its content cvCreateImage cvCreateImageHeader cvReleaseImageHeader cvReleaseImage cvCreateImageData cvReleaseImageData cvSetImageROI cvCopyImage cvCloneImage cvGetImageRawData

#include ”ipl.h” #include “cv.h” #include “cxcore.h” … IplImage *img = cvCreateImage(cvSize(150, 100), IPL_DEPTH_8U, 3); /////// Use the image //////// cvReleaseImage(&img); Remark: In some cases (when we work with images captured from the camera), would be more convenient to initialize the IplImage structure using the IPL functions.

Arithmetical/Logical operations Arithmetical operations: unary; cvAddS, cvSubS, ... binary: cvAdd, cvMul, cvSub, cvMatMulAdd, cvInvert, ... Logical operations unary: cvAndS, cvOrS, cvXorS, ... binary: cvAnd, cvOr, cvXor, ...

Remark: Most of the OpenCV functions are defined to support both the IplImage and CvMat data types. That’s possible because of the definition of CvArr data type: typedef void CvArr; The OpenCV functions look-up for the first integer of the structure being passed, in order to distinguish between the two data types. In the case of IplImage, this integer is equal to the size of IplImage structure, meanwhile it is equal to 0x4224xxxx in the case of CvMat.

Morphological operations Image filtering Based on convolution with fixed kernel: cvLaplace, cvSobel, cvSmooth Morphological operations cvErode, cvDilate cvMorphologyEx (advanced operations: opening, closing, Top-Hat, etc.) user-defined structuring elements: cvCreateStructuringElementEx cvReleaseStructuringElement

Example: application of the erosion function  

Color space conversion cvCvtColor allows the following color-space conversions: CV_RGB2GRAY CV_RGB2HSV CV_RGB2YCrCb Histogram cvCreateHist, cvReleaseHist cvCalcHist, cvCopyHist cvCompareHist, cvThreshHist cvGetMinMaxHistValue, cvNormalizeHist

Linear/geometrical transformations cvFFT (Fast Fourier Transform) cvDCT (Discrete Cosine Transform) cvResize, cvMirror, cvConvertScale Feature extraction cvCanny (border detection) cvHoughLines (line detection) cvFindCornerSubPix (corner detection)

Example of border detection  

Image statistics Drawing functions cvNorm (C- , L1- and L2-norm) cvMoments (spatial and central moments) cvMinMaxLoc (find the min/max values) Drawing functions cvLine, cvRectangle cvCircle, cvEllipse cvPolyLine, cvFillPoly cvInitFont, cvPutText

Motion analysis estimators: Kalman, Condensation cvKalmanXX, cvCondensXX movement patterns cvCalcMotionGradient, cvMotionHistoryUpdate optical flow cvCalcOpticalFlowPyrLK (implements the Lucas-Kanade method based on pyramidal decomposition) tracking cvMeanShift, cvCamShift, cvSnakeImage background substraction

3D reconstruction - camera calibration cvCalibrateCamera, cvFindExtrinsecCameraParams, cvUnDistort hand detection cvFindHandRegion pose estimation cvPOSIT finding pixel correspondence in a pair of stereo images cvFindStereoCorrespondence

Example of distorsion correction

Object detection (faces) The object detection algorithm implemented in OpenCV is based on the following papers: - Paul Viola and Michael J. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE CVPR, 2001 and - Rainer Lienhart and Jochen Maydt. An Extended Set of Haar-like Features for Rapid Object Detection. IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep. 2002.

Idea A classifier (namely a cascade of boosted classifiers working with haar-like features) is trained with a few hundreds of sample views of a particular object (i.e., a face or a car), called positive examples, that are scaled to the same size (say, 20x20), and negative examples - arbitrary images of the same size.

The weak classifier outputs a "1" if the region is likely to show the object of interest (i.e., face/car), and "0" otherwise - The classifier is designed so that it can be easily "resized" in order to be able to find the objects of interest at different sizes, which is more efficient than resizing the image itself. - The word "cascade" in the classifier name means that the resultant classifier consists of several simpler classifiers (stages) that are applied subsequently to a region of interest until at some stage the candidate is rejected or all the stages are passed.

Data structures and functions implemented in OpenCV for face detection CvHaarClassifierCascade (structure representing a cascade of classifiers) cvLoadHaarClassifierCascade (reads from a file a cascade of classifiers and stores it in the CvHaarClassifierCascade structure). The cascade is stored in a XML file format: C:\Archivos de programa\Intel\OpenCV\data\haarcascades\...) cvHaarDetectObjects (detects the objects in the image) cvReleaseHaarClassifierCascade (frees the memory occupied by the CvHaarClassifierCascade structure)

HighGUI library Allows a fast and easy interaction/visualization with/of images - Reading an image from a file cvLoadImage (const char* filename, int iscolor CV_DEFAULT(1)); Writing an image to a file cvSaveImage (const char* filename, const CvArr* image); It supports several formats: BMP, GIF, JPG, TIFF, etc. Open a window for visualization - cvNamedWindow (const char* name, int flags); Visualize the image cvShowImage (const char* name, const CvArr* image);

The structure of OpenCV The ‘cv.h’ contains: general image processing functions: filter, color conversion, morphological operators, structural analysis, motion analysis, pattern recognition (object detection), camera calibration and 3D reconstruction The ‘cxcore.h’ contains: basic structures, arithmetical/logical operators (copy, transformation), dynamic structures (sets, graphs, trees), drawing functions, error handling and system functions The ‘cvaux.h’ - stereo correspondence, texture descriptors, 2D-3D trackers, background segmentation, morphing, etc. The ‘highgui.h’ contains: user interface

Installation of IPL/OpenCV libraries and environment settings for MSVC++ In order to keep track with the following settings, OpenCV folder and IPL folder (named ‘plsuite’) must be installed in C:\Archivos de Programa\Intel\ in the ‘c’ source file add the header files needed: ‘ipl.h’, ‘cv.h’, ‘cxcore.h’, ‘highgui.h’, ‘cvaux.h’, ‘cvhaartraining.h’ from the ‘Project’ menu, choose ‘Settings’ and click on the ‘Link Tab’. In the field: ‘Object/library modules’ add the following: ipl.lib cv.lib cxcore.lib highgui.lib cvaux.lib cvhaartraining.lib (if you use the face detector functions)

from the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the field ‘Show directories for’ choose ‘Include files’, then edit the following paths: C:\ARCHIVOS DE PROGRAMA\INTEL\PLSUITE\INCLUDE C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\CV\INCLUDE C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\OTHERLIBS\HIGHGUI C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\CXCORE\INCLUDE C:\ ARCHIVOS DE PROGRAMA\INTEL\OPENCV\apps\HaarTraining\include (if you are using the face detector)

- from the menu ‘Tools’ choose ‘Options’ and then click on the ‘Directories’ tab. In the field ‘Show directories for’ choose ‘Library files’, then edit the following paths: C:\ARCHIVOS DE PROGRAMA\INTEL\PLSUITE\LIB\MSVC C:\ARCHIVOS DE PROGRAMA\INTEL\OPENCV\LIB

the last step consists in adding the paths for DLLs in the ‘Environment Variables’ from ‘Control Panel’, choose ‘System’, then ‘Advanced’ and finally click on ‘Environment Variable’. From dialog box that appears, in the ‘System Variables’ section, click and edit the ‘PATH’ item you have to add the following route: C:\Archivos de programa\Intel\OpenCV\bin C:\Archivos de programa\Intel\plsuite\bin

Examples More information... Three examples are included (provided in separate files): first is about getting a live stream from a webcam (you must make sure you have DirectX library installed second is about detecting faces in an image by using the detector which comes with the OpenCV library third is about image convolution More information... http://www.cvc.uab.es/~bogdan/CV/cv.html IPL/OpenCV online documentation (html, pdf files) C:\Archivos de programa\Intel\OpenCV\samples http://www.site.uottawa.ca/~laganier/tutorial/opencv+directshow/cvision.htm http://groups.yahoo.com/group/OpenCV/ (you have to register)