1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Image Features Kenton McHenry, Ph.D. Research Scientist

2. Raster Images 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 [Hoiem, 2012] image(234, 452) = 0.58

3. Neighborhoods of Pixels For nearby surface points most factors do not change much Local differences in brightness [Hoiem, 2012]

4. Features Interest points Points or collections of points that are somehow relevant to identifying the contents of an image Subset of the image Far less data than the total number of pixels Repeatability A feature detector should be likely to detect the feature in a scene under a variety of lighting, orientation, and other variable conditions.

5. Features Edges Corners Blobs

6. Edges Changes in intensity Indicative of changes in albedo Indicative of changes in orientation Indicative of changes in distance How do we find such regions?

7. Matlab Commercial numerical computing environment Good at matrix operations Images are matrices Great for quickly prototyping ideas Image Processing Toolbox

8. Matlab >ls >cd Matlab >ls >whos >I = imread(‘mushrooms.jpg’); >whos >Ig = mean(I, 3); >whos >Ig >image(Ig); >colormap gray >imagesc(Ig);

9. Matlab >I = imread(‘intensities1.png’); >Ig = avg(I, 3); >imagesc(Ig); >surf(Ig); >axis vis3d; >rotate3d on

10. How do we find patterns in a Raster Image? Geometry flashback Vectors 2D: (x,y) A magnitude and direction > a = [2, 1]; > x = a(1); > y = a(2); > length = sqrt(x^2+y^2) > length = norm(a); > direction = a / norm(a); Real numbers are called scalars (2,1)

11. How do we find patterns in a Raster Image? Vector operations With scalars: add, subtract multiply, divide With vectors: cross product: gets the angle between two vectors dot product: projects one vector into another http://en.wikipedia.org/wiki/Dot_product

12. How do we find patterns in a Raster Image? > a = [2,1] > b = [3,2] > a(1)*b(1) + a(2)*b(2) > dot(a,b) > dot(a/norm(a),b/norm(b)) > plot(a(1),a(2),’b.’); > plot([0,a(1)],[0,a(2)],’b-’); > hold on; > plot([0,b(1)],[0,b(2)],’r-’);

13. How do we find patterns in a Raster Image? > a = [2,1] > b = [-1,2] > dot(a/norm(a),b/norm(b)) > dot(a/norm(a),a/norm(a)) > b = [-2,-1] > dot(a/norm(a),b/norm(b))

14. How do we find patterns in a Raster Image? Dot product gives values near 1 for similar vectors. Not restricted to two dimensions 0.920.93 0.950.89 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 0.570.37 0.500.60 a=[0.92,0.93,0.95,0.89]b=[0.57,0.37,0.50,0.60]

15. How do we find patterns in a Raster Image? Use dot product with an image of the target e.g. vertical edges Use dot product at every possible location using a target:

16. Convolution Given functions f(x) and g(x): In a discrete image this is the process of calculating the dot product at every location in an image using a target image (called a filter). http://en.wikipedia.org/wiki/Convolution

17. Clarification: Continuous vs. Discrete Continuous: smooth, no sharp discontinuities Discrete: distinct separate values Sampling: going from continuous to discrete by measuring over a small interval

18. Clarification: Continuous vs. Discrete

19. Convolution >Ig = mean(imread(‘stripes1.png’), 3); >filter = [-1 1; -1 1] >imagesc(filter); >filter = filter / norm(filter); >Ig = Ig / norm(Ig); >responses = abs(conv2(Ig, filter)); >filter = [1 1; -1 -1]; >filter = filter / norm(filter); >responses = abs(conv2(Ig, filter));

20. So how do I get all edges? Not so fast, we have been dealing with perfect artificial images… Real images are noisy! [Zoom Demo]

21. Noise >Ig = mean(imread(‘stripes2.png’), 3); >Ig = Ig / norm(Ig); >filter = [-1 1; -1 1] >filter = filter / norm(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

22. Smoothing Assume noise is small and infrequent Use neighbors values as a guide to remove noise

23. Smoothing >Ig = mean(imread(‘stripes2.png’), 3); >Ig = Ig / norm(Ig); >filter = [1 1; 1 1] >filter = filter / norm(filter); >imagesc(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir) >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

24. Smoothing Weight so that nearby points are more important Gaussian function http://en.wikipedia.org/wiki/Gaussian_function

25. Smoothing >filter = fspecial(‘guassian’, 50, 5); >filter = filter / norm(filter); >imagesc(filter); >surf(filter) >axis vis3d >rotate3d on

26. Smoothing >Ig = mean(imread(‘stripes2.png’), 3); >Ig = Ig / norm(Ig); >filter = fspecial(‘guassian’, 8, 2); >filter = filter / norm(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir) >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

27. Smoothing and Edges Calculus flashback Derivative: Rate of changes along function Large derivatives indicate sharp changes Edges in an image Derivative of convolved function is equal to convolution with derivative of one of the two functions

28. Smoothing and Edges Gaussian Derivative:

29. Smoothing and Edges Finite differences Approximation of derivative in discrete space Subtract neighbors

30. Smoothing and Edges 2D Guassian Partial derivatives along x and y direction

31. Smoothing and Edges >gaussian = fspecial(‘guassian’, 50, 5); >guassian = filter / norm(filter); >filter = diff(gaussian, 1, 1); >imagesc(filter); >surf(filter) >axis vis3d >rotate3d on >filter = diff(gaussian, 1, 2); >imagesc(filter); >surf(filter)

32. Smoothing and Edges >Ig = mean(imread(‘stripes2.png’), 3); >Ig = Ig / norm(Ig); >filter = fspecial(‘guassian’, 8, 2); >filter = diff(filter, 1, 2); >filter = filter / norm(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

33. So now how do I get all edges? Canny edge detector [Canny, 86] Generate two filters one derivate in x and one in y Convolve image with each filter Use normalized responses in x and y direction to determine overall strength and angle Thresholding to get strong long thin edges

34. Smoothing and Edges >Ig = mean(imread(‘stripes2.png’), 3); >Ig = Ig / norm(Ig); >gaussian = fspecial(‘guassian’, 8, 2); >fx = diff(gaussian, 1, 2); >fx = fx / norm(fx); >fy = diff(gaussian, 1, 1); >fy = fy / norm(fy); >Igx = abs(conv2(Ig, fx, ‘same’)); >Igy = abs(conv2(Ig, fy, ‘same’)); >imagesc(Igx) >imagesc(Igy) >imagesc(Igx + Igy); >imagesc((Igx + Igy)>100);

35. Edges >Ig = mean(imread(‘mushrooms.jpg’), 3); >edge(Ig, ‘canny’, 0.01); >edge(Ig, ‘canny’, 0.1); >edge(Ig, ‘canny’, 0.5);

36. Edges [iPad Demo]

37. Lines Hough Transform

38. Lines 00000000 00000000 00001000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 (x,y) = (350,200) 0350 30403 60348 90200 120-2 150-203 180-350 ……  r 30 350

39. Lines Threshold votes to identify lines [Demo] http://en.wikipedia.org/wiki/Hough_transform

40. Lines Notice these are infinite lines and not line segments Extra work to find endpoints Can be used for many parametric shapes e.g. circles, r 2 = (x 2 - a 2 ) + (y 2 - b 2 )

41. Corners Harris detector [Harris, 1988] Partial derivatives in intensity Averaged over a small window http://en.wikipedia.org/wiki/Corner_detection

42. Corners IxIxIxIx IyIyIyIy

43. [Java Demo] [iPad Demo]

44. Blobs Lets treat raster images as 2D continuous functions again Gradient: Vector that points in the direction of greatest increase Think intensity again Divergence: The magnitude of changes in a vector field In our image the vector field is the gradient calculated at every pixel Laplacian: The divergence of a functions gradient http://homepages.inf.ed.ac.uk/rbf/HIPR2/log.htm

45. Why do we care about the Laplacian? Indicates large changes in intensity when applied to an image Edges >Ig = mean(imread(‘mushrooms.jpg’), 3); >Ig = Ig / norm(Ig); >filter = fspecial(‘log’, 8, 2); >filter = filter / norm(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

46. Why do we care about the Laplacian? >filter = fspecial(‘log’, 50, 5); >imagesc(filter) >surf(filter) >axis vis3d Again not really taking Laplacian of the image but of the Gaussian function used to smooth the image http://homepages.inf.ed.ac.uk/rbf/HIPR2/log.htm

47. Back to blobs Recall how convolution in discrete space involves the dot product which found vectors (i.e. patterns) that looked like the filter The Laplacian of a Gaussian looks like a circular blob

48. Blobs >Ig = mean(imread(‘mushrooms.jpg’), 3); >Ig = Ig / norm(Ig); >filter = fspecial(‘log’, 100, 20); >filter = filter / norm(filter); >Ir = abs(conv2(Ig, filter)); >imagesc(Ir)

49. Blobs [Java Demo]

50. Superpixels Create regions of similar color (and other properties) Graph based formulations Treat pixels as vertices Create edges among neighboring pixels Weight edges by how similar the pixels they connect are

51. Superpixels

53. Cut edges according to specified criteria: [Shi, 2000] Remove edges with low weight Maximize homogeniety of connected components Maintain a minimum required size …

54. Superpixels

55. Merges edges according to specified criteria: [Felzenswalk, 2004] Weight = Distance + k*Size Distance: Treat RGB values as a 3D point and use distance Size: Number of pixels in regions thus far Maintain a certain size of regions

56. Superpixels

57. [Java Demo]

58. Conclusion Features Interest points, Repeatable, Subset of image Edges Lines Circles … Corners Blobs Superpixels

59. Image and Spatial Data Analysis Group http://isda.ncsa.illinois.edu Questions?