Software Resources Multimodal Interaction Dr. Mike Spann m.spann@bham.ac.uk http://www.eee.bham.ac.uk/spannm

Contents Introduction ImageJ introduction Image representation in ImageJ Displaying images in ImageJ ImageJ utility class Regions of interest Writing plug-ins OpenCV introduction Image handling and display in OpenCV Other libraries Key tips Key resources

Introduction Two free platforms for developing image processing software are ImageJ and OpenCV ImageJ – Java OpenCV – C++, C# They both allow rapid development of image processing algorithms ImageJ through the use of plugins Both have extensive libraries of well known image processing functions for the user to build upon

ImageJ introduction All of this material is taken from the ImageJ tutorial accessed from my web-site http://mtd.fh-hagenberg.at/depot/imaging/imagej/ Also check out the ImageJ home page http://rsb.info.nih.gov/ij/ ImageJ is a free image processing system allowing easy development of Java based image processing algorithms in the form of plug-ins

ImageJ introduction It comes with a user-friendly GUI and can run either as an application or an applet http://rsb.info.nih.gov/ij/applet/

ImageJ introduction It can handle 8,16,24 and 32 bit images It can handle most standard image formats TIFF GIF JPEG BMP DICOM It can handle stacks of images Also there are plug-ins allowing it to handle movie files It can also handle regions of interest (ROI’s)

ImageJ introduction The key point about ImageJ is that it is simple to add your own algorithms (written as plug-ins) callable from the front-end GUI File I/O taken care of by ImageJ Pixel access easy from image handles defined within ImageJ

Image representation in ImageJ ImageJ has 5 built-in image classes 8 bit grayscale (byte) 8 bit colour (byte) 16 bit grayscale (short) RGB colour (int) 32 bit image (float) It also supports image stacks consisting of images (slices) of the same size

Image representation in ImageJ ImageJ uses 2 classes to represent and manipulate images ImagePlus An image is represented by an ImagePlus object ImageProcessor This holds the pixel data and contains methods to access the pixel data

Image representation in ImageJ Pixel access methods in ImageProcessor include Object getPixels() – returns a reference to the pixel array (need to cast to appropriate type) int getHeight() – returns height of pixel array int getWidth() – returns width of pixel array

Image representation in ImageJ A subclass of ImageProcessor is passed to the run() method of the plug-in filter (see later) ByteProcessor - 8 bit grayscale images ShortProcessor – 16 bit grayscale images ColorProcessor – RGB images FloatProcessor – 32 bit floating point images

Image representation in ImageJ Pixel representation in ImageJ uses the byte datatype for grayscale and colour images and short for 16-bit grayscale byte/short are signed data types byte ranges from –128 to 127 short ranges from –32768 to 32767 Obviously grayscale values are usually positive values

Image representation in ImageJ To cast a byte to an integer, we need to eliminate the sign bit Can cast back the other way easily enough byte[] pixels=(byte[]) ip.getPixels(); int grey=0xxff & pixels[j]; pixels[j]=(byte) grey;

Image representation in ImageJ The ColorProcessor return the pixels as int[] and the RGB values are packed into the one int variable 31 0 bit int[] pixels=(int[]) ip.getPixels(); int red=(0xxff0000 & pixels[j]) >> 16; int green=(0xx00ff00 & pixels[j]) >> 8; int blue=(0xx0000ff & pixels[j]);

Image representation in ImageJ Can reconstitute an RGB array by shifting the other way : pixels[j]=((red & 0xff)<<16)+((green & 0xff)<<8)+(blue & 0xff);

Displaying images in ImageJ A class ImageWindow is used to display ImagePlus objects We don’t normally need to access methods of ImageWindow These are automatically called from ImagePlus methods show(), draw() and updateAndDraw()

ImageJ utility class ImageJ contains a class called IJ which contains a number of useful static methods Error messages static void error(String message) – displays an error message in a dialog box static boolean showMessageWithCancel(String title, String message) – allows the user to cancel the plug in or continue

ImageJ utility class Displaying text Displaying text in a status bar static void write(String s) - Outputs text in a window – useful for displaying textual or numerical results of algorithms Displaying text in a status bar static void showStatus(String s)

ImageJ utility class User input static double getNumber(String prompt, double default) – Allows the user to input a number in a dialog box static String getString(String prompt, String default) – Allows the user to input a string in a dialog box The GenericDialog class is a more sophisticated way of inputting more than a single number or string

Regions of interest (ROI’s) A plug in filter does not always have to work on the whole image ImageJ supports ROI’s which can are usually rectangular but can be other shapes as well. We set/get the ROI using the following method of the ImageProcessor class void setROI(int x, int y, int w int h) Rectangle getROI()

Writing plug-ins To write a plug in requires developing a class which implements either the PlugIn or PlugInFilter interfaces The second is more usual as this is used when the filter requires an input image import ij.*; import ij.plugin.filter.PlugInFilter; import ij.process; class myPlugin implements PlugInFilter

Writing plug-ins Methods setup() and run() must then be provided Method setup() sets up the plugin filter for use String arg allows input arguments to be passed to the plugin Argument imp handled automatically by ImageJ It’s the currently active image int setup(String arg, ImagePlus imp)

Writing plug-ins Method setup() returns a flag word representing the capabilities of the plug-in filter (for example the types of images it can handle). For example : static int DOES_8G static int DOES_RGB static int DOES_ALL static int NO_CHANGES (plug in doesn’t change the image data) static int ROI_REQUIRED etc

Writing plug-ins The run() method is called when the plugin is run from the ImageJ menu It contains the code to process individual pixels If no input image is required If an input image is required void run(String arg) void run(ImageProcessor ip)

Writing plug-ins Once the Java program is written and compiled, the .class file is stored in the plug-in directory which is a sub-directory of the ImageJ home directory The class name should contain an underscore symbol (eg MyAlgorithm_ ) It then appears under the plug-in sub-menu of the ImageJ gui

OpenCV introduction Based on Intel IP Library More than 300 functions Features Multi-platform Provides a simple window manager with sliders mouse call backs etc Uses Intel Integrated Performance Primitives to enhance performance Tutorials & Documentation Sample code Download from sourceforge.net/projects/opencvlibrary/ Operates under Windows 95/98/NT/2000/XP POSIX Linux/BSD/OSX/Win2K/WinXP

Image handling and display in OpenCV Still images from file Using cvLoadImage() IplImage* cvLoadImage(const char* filename,int iscolor=1); Returns a pointer to an ipl image Filename: the name of the image file iscolor: >0 image is loaded as 3 channel colour 0 image is loaded as grey-level <0 No. of channels determined by file File formats Windows bitmaps: bmp, DIB JPEG files: jpeg, jpg, jpe Portable Network Graphics: png Portable image format: pbm, pgm, ppm Sun raster: sr, ras Tiff Files: tiff, tif

Image handling and display in OpenCV Simple code to load and display an image Uses library HighGUI Need to register image window and display it int cvNamedWindow(const char* name, int flags); name: id name and text on image banner int: 1, autosize to fit the image void cvShowImage(const char* name, const CvArr* img); name: id name and text on image banner img: pointer to image to be displayed CvCapture* cvCaptureFromFile(const char* fname); fname: video file name CvCapture* cvCaptureFromCAM(int index); index: positive integer specifies video source

/* usage: prog <image_name> */ #include "cv.h“ /* OpenCV data types and prototypes */ #include "highgui.h“ /* Image read and display*/ int main( int argc, char** argv ) {IplImage* img; if(argc==2 && (img=cvLoadImage(argv[1], 1)) !=0) {cvNamedWindow("Image view", 1); cvShowImage("Image view", img); cvWaitKey(0); /*very important, contains event */ /*processing loop inside */ cvDestroyWindow("Image view"); cvReleaseImage(&img); return 0; } return -1;

Other libraries:(CV) Image Processing Structural analysis Edges, filters, morphologic operators, pyramids, histograms, boundary tracing Structural analysis Contour processing, computational geometry Motion Analysis and Object Tracking Object tracking, optical flow, motion templates Pattern Recognition Camera calibration and 3D Reconstruction Camera calibration, pose estimation, epipolar geometry

Other libraries:cxcore Operations on arrays Linear algebra, statistics, random number generator Dynamic structures Trees, graphs, sets, sequences Drawing Curves, text, point sets, contours

Other libraries:ML Machine Learning Bayes classifier K-nearest neighbour classifier Support Vector Machine Decision Trees Boosting Random Forest Expectation Maximisation Neural Networks

Other libraries:CvCam/Cvaux Video Input / Output Select and set up a camera Render a video stream Control the Video Process video frames Multiple cameras under Linux and Windows Cvaux Experimental functions

Key tips IplImage data Types IPL_DEPTH_8U unsigned 8 bit IPL_DEPTH_8S signed 8 bit IPL_DEPTH_16U unsigned 16 bit IPL_DEPTH_16S signed 16 bit IPL_DEPTH_32S signed 32 bit IPL_DEPTH_32F single precision floating point IPL_DEPTH_ 64F double precision floating point Channels: 1, 2, 3 or 4 sequence is: b0, g0, r0, b1, g1, r1

Key tips Accessing image pixels Coordinates are indexed from 0 Origin is top left or bottom left To access an element of an 8-bit 1-channel image, I (IplImage* img): I(x,y)~ ((uchar*)(img->imageData + img->widthStep*y)[x]

Key tips To access an element of an 8-bit 3-channel image, I (IplImage* img): I(x,y)blue~ ((uchar*)(img->imageData + img->widthStep*y)[x*3] I(x,y)green~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+1] I(x,y)red~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+2] To access an element of a 32-bit floating point 1-channel image, I (IplImage* img): I(x,y)~ ((float*)(img->imageData + img->widthStep*y)[x]

Key tips In general to access an element of an N-channel image of type T I(x,y)c~ ((T*)(img->imageData + img->widthStep*y)[x*N+c] More simply using the provided macro CV_IMAGE_ELEM(img, T, y, x*N+c) Efficient way to increment brightness of point (100, 100) by 30 CvPoint pt = {100, 100}; uchar *tmp_ptr=&((uchar*)(img->ImageData + img->widthStep*pt.y))[x*3]; tmp_ptr[0]+=30; tmp_ptr[1]+=30; tmp_ptr[2]+=30;

Key resources A brief intro to OpenCV: www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/index.html A more detailed guide (also describes DirectX/DirectShow): http://www.site.uottawa.ca/~laganier/tutorial/ opencv+directshow/cvision.htm Wikipedia http://en.wikipedia.org/wiki/OpenCV Book Learning OpenCV: Computer Vision with the OpenCV Library (Paperback) Gary Bradski and Adrian Kaehler O'Reilly Media, Inc. (15 Jul 2008) ISBN 0596516134