1. Software Resources Multimodal Interaction Dr. Mike Spann

2. Contents Introduction ImageJ introduction
Image representation in ImageJ
Displaying images in ImageJ
ImageJ utility class
Regions of interest
Writing plug-ins
OpenCV introduction
Image handling and display in OpenCV
Other libraries
Key tips
Key resources

3. Introduction
Two free platforms for developing image processing software are ImageJ and OpenCV
ImageJ – Java
OpenCV – C++, C#
They both allow rapid development of image processing algorithms
ImageJ through the use of plugins
Both have extensive libraries of well known image processing functions for the user to build upon

4. ImageJ introduction
All of this material is taken from the ImageJ tutorial accessed from my web-site
Also check out the ImageJ home page
ImageJ is a free image processing system allowing easy development of Java based image processing algorithms in the form of plug-ins

5. ImageJ introduction
It comes with a user-friendly GUI and can run either as an application or an applet

6. ImageJ introduction It can handle 8,16,24 and 32 bit images
It can handle most standard image formats
TIFF
GIF
JPEG
BMP
DICOM
It can handle stacks of images
Also there are plug-ins allowing it to handle movie files
It can also handle regions of interest (ROI’s)

7. ImageJ introduction
The key point about ImageJ is that it is simple to add your own algorithms (written as plug-ins) callable from the front-end GUI
File I/O taken care of by ImageJ
Pixel access easy from image handles defined within ImageJ

8. Image representation in ImageJ
ImageJ has 5 built-in image classes
8 bit grayscale (byte)
8 bit colour (byte)
16 bit grayscale (short)
RGB colour (int)
32 bit image (float)
It also supports image stacks consisting of images (slices) of the same size

9. Image representation in ImageJ
ImageJ uses 2 classes to represent and manipulate images
ImagePlus
An image is represented by an ImagePlus object
ImageProcessor
This holds the pixel data and contains methods to access the pixel data

10. Image representation in ImageJ
Pixel access methods in ImageProcessor include
Object getPixels() – returns a reference to the pixel array (need to cast to appropriate type)
int getHeight() – returns height of pixel array
int getWidth() – returns width of pixel array

11. Image representation in ImageJ
A subclass of ImageProcessor is passed to the run() method of the plug-in filter (see later)
ByteProcessor - 8 bit grayscale images
ShortProcessor – 16 bit grayscale images
ColorProcessor – RGB images
FloatProcessor – 32 bit floating point images

12. Image representation in ImageJ
Pixel representation in ImageJ uses the byte datatype for grayscale and colour images and short for 16-bit grayscale
byte/short are signed data types
byte ranges from –128 to 127
short ranges from –32768 to 32767
Obviously grayscale values are usually positive values

13. Image representation in ImageJ
To cast a byte to an integer, we need to eliminate the sign bit
Can cast back the other way easily enough
byte[] pixels=(byte[]) ip.getPixels();
int grey=0xxff & pixels[j];
pixels[j]=(byte) grey;

14. Image representation in ImageJ
The ColorProcessor return the pixels as int[] and the RGB values are packed into the one int variable 31
0 bit
int[] pixels=(int[]) ip.getPixels();
int red=(0xxff0000 & pixels[j]) >> 16;
int green=(0xx00ff00 & pixels[j]) >> 8;
int blue=(0xx0000ff & pixels[j]);

15. Image representation in ImageJ
Can reconstitute an RGB array by shifting the other way :
pixels[j]=((red & 0xff)<<16)+((green & 0xff)<<8)+(blue & 0xff);

16. Displaying images in ImageJ
A class ImageWindow is used to display ImagePlus objects
We don’t normally need to access methods of ImageWindow
These are automatically called from ImagePlus methods show(), draw() and updateAndDraw()

17. ImageJ utility class
ImageJ contains a class called IJ which contains a number of useful static methods
Error messages
static void error(String message) – displays an error message in a dialog box
static boolean showMessageWithCancel(String title, String message) – allows the user to cancel the plug in or continue

18. ImageJ utility class Displaying text Displaying text in a status bar
static void write(String s) - Outputs text in a window – useful for displaying textual or numerical results of algorithms
Displaying text in a status bar
static void showStatus(String s)

19. ImageJ utility class User input
static double getNumber(String prompt, double default) – Allows the user to input a number in a dialog box
static String getString(String prompt, String default) – Allows the user to input a string in a dialog box
The GenericDialog class is a more sophisticated way of inputting more than a single number or string

20. Regions of interest (ROI’s)
A plug in filter does not always have to work on the whole image
ImageJ supports ROI’s which can are usually rectangular but can be other shapes as well.
We set/get the ROI using the following method of the ImageProcessor class
void setROI(int x, int y, int w int h)
Rectangle getROI()

21. Writing plug-ins
To write a plug in requires developing a class which implements either the PlugIn or PlugInFilter interfaces
The second is more usual as this is used when the filter requires an input image
import ij.*;
import ij.plugin.filter.PlugInFilter;
import ij.process;
class myPlugin implements PlugInFilter

22. Writing plug-ins Methods setup() and run() must then be provided
Method setup() sets up the plugin filter for use
String arg allows input arguments to be passed to the plugin
Argument imp handled automatically by ImageJ
It’s the currently active image
int setup(String arg, ImagePlus imp)

23. Writing plug-ins
Method setup() returns a flag word representing the capabilities of the plug-in filter (for example the types of images it can handle). For example :
static int DOES_8G
static int DOES_RGB
static int DOES_ALL
static int NO_CHANGES (plug in doesn’t change the image data)
static int ROI_REQUIRED
etc

24. Writing plug-ins
The run() method is called when the plugin is run from the ImageJ menu
It contains the code to process individual pixels
If no input image is required
If an input image is required
void run(String arg)
void run(ImageProcessor ip)

25. Writing plug-ins
Once the Java program is written and compiled, the .class file is stored in the plug-in directory which is a sub-directory of the ImageJ home directory
The class name should contain an underscore symbol (eg MyAlgorithm_ )
It then appears under the plug-in sub-menu of the ImageJ gui

26. OpenCV introduction Based on Intel IP Library More than 300 functions
Features
Multi-platform
Provides a simple window manager with sliders mouse call backs etc
Uses Intel Integrated Performance Primitives to enhance performance
Tutorials & Documentation
Sample code
Download from
sourceforge.net/projects/opencvlibrary/
Operates under
Windows 95/98/NT/2000/XP
POSIX Linux/BSD/OSX/Win2K/WinXP

27. Image handling and display in OpenCV
Still images from file
Using cvLoadImage()
IplImage* cvLoadImage(const char* filename,int iscolor=1);
Returns a pointer to an ipl image
Filename: the name of the image file
iscolor: >0 image is loaded as 3 channel colour
0 image is loaded as grey-level
<0 No. of channels determined by file
File formats
Windows bitmaps: bmp, DIB
JPEG files: jpeg, jpg, jpe
Portable Network Graphics: png
Portable image format: pbm, pgm, ppm
Sun raster: sr, ras
Tiff Files: tiff, tif

28. Image handling and display in OpenCV
Simple code to load and display an image
Uses library HighGUI
Need to register image window and display it
int cvNamedWindow(const char* name, int flags); name: id name and text on image banner int: 1, autosize to fit the image
void cvShowImage(const char* name, const CvArr* img); name: id name and text on image banner img: pointer to image to be displayed
CvCapture* cvCaptureFromFile(const char* fname); fname: video file name
CvCapture* cvCaptureFromCAM(int index); index: positive integer specifies video source

29. /* usage: prog <image_name> */
#include "cv.h“ /* OpenCV data types and prototypes */
#include "highgui.h“ /* Image read and display*/
int main( int argc, char** argv )
{IplImage* img;
if(argc==2 && (img=cvLoadImage(argv[1], 1)) !=0)
{cvNamedWindow("Image view", 1);
cvShowImage("Image view", img);
cvWaitKey(0); /*very important, contains event */
/*processing loop inside */
cvDestroyWindow("Image view");
cvReleaseImage(&img);
return 0; }
return -1;

30. Other libraries:(CV) Image Processing Structural analysis
Edges, filters, morphologic operators, pyramids, histograms, boundary tracing
Structural analysis
Contour processing, computational geometry
Motion Analysis and Object Tracking
Object tracking, optical flow, motion templates
Pattern Recognition
Camera calibration and 3D Reconstruction
Camera calibration, pose estimation, epipolar geometry

31. Other libraries:cxcore
Operations on arrays
Linear algebra, statistics, random number generator
Dynamic structures
Trees, graphs, sets, sequences
Drawing
Curves, text, point sets, contours

32. Other libraries:ML Machine Learning Bayes classifier
K-nearest neighbour classifier
Support Vector Machine
Decision Trees
Boosting
Random Forest
Expectation Maximisation
Neural Networks

33. Other libraries:CvCam/Cvaux
Video Input / Output
Select and set up a camera
Render a video stream
Control the Video
Process video frames
Multiple cameras under Linux and Windows
Cvaux
Experimental functions

34. Key tips IplImage data Types IPL_DEPTH_8U unsigned 8 bit
IPL_DEPTH_8S signed 8 bit
IPL_DEPTH_16U unsigned 16 bit
IPL_DEPTH_16S signed 16 bit
IPL_DEPTH_32S signed 32 bit
IPL_DEPTH_32F single precision floating point
IPL_DEPTH_ 64F double precision floating point
Channels: 1, 2, 3 or 4 sequence is: b0, g0, r0, b1, g1, r1

35. Key tips Accessing image pixels Coordinates are indexed from 0
Origin is top left or bottom left
To access an element of an 8-bit 1-channel image, I (IplImage* img):
I(x,y)~ ((uchar*)(img->imageData + img->widthStep*y)[x]

36. Key tips
To access an element of an 8-bit 3-channel image, I (IplImage* img): I(x,y)blue~ ((uchar*)(img->imageData + img->widthStep*y)[x*3] I(x,y)green~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+1] I(x,y)red~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+2]
To access an element of a 32-bit floating point 1-channel image, I (IplImage* img): I(x,y)~ ((float*)(img->imageData + img->widthStep*y)[x]

37. Key tips
In general to access an element of an N-channel image of type T I(x,y)c~ ((T*)(img->imageData + img->widthStep*y)[x*N+c]
More simply using the provided macro CV_IMAGE_ELEM(img, T, y, x*N+c)
Efficient way to increment brightness of point (100, 100) by 30 CvPoint pt = {100, 100}; uchar *tmp_ptr=&((uchar*)(img->ImageData img->widthStep*pt.y))[x*3]; tmp_ptr[0]+=30; tmp_ptr[1]+=30; tmp_ptr[2]+=30;

38. Key resources
A brief intro to OpenCV:
A more detailed guide (also describes DirectX/DirectShow): opencv+directshow/cvision.htm
Wikipedia
Book Learning OpenCV: Computer Vision with the OpenCV Library (Paperback) Gary Bradski and Adrian Kaehler O'Reilly Media, Inc. (15 Jul 2008) ISBN