1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Image Feature Descriptors Kenton McHenry, Ph.D. Research Scientist

2. Raster Images 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 [Hoiem, 2012] image(234, 452) = 0.58

3. Neighborhoods of Pixels For nearby surface points most factors do not change much Local differences in brightness [Hoiem, 2012]

4. Features Interest points Points or collections of points that are somehow relevant to identifying the contents of an image Subset of the image Far less data than the total number of pixels Repeatability A feature detector should be likely to detect the feature in a scene under a variety of lighting, orientation, and other variable conditions. Descriptors A relatively short numerical representation that can be used to identify a specific feature instance

5. Descriptors, Why? Matching: Match an object in two images based on similar features 3D Reconstruction: Stereopsis Tracking: Follow an object in a video by following its features Object Recognition: Find objects based on known features it will posses Segmentation: Break an image up into more meaningful regions based on the seen features.

6. Feature Descriptors

7. Feature Descriptors for Edges Lines: (m, b) (r,  ) Circles: r 2 = (x 2 - a 2 ) + (y 2 - b 2 ) (a, b, r) Other shapes Follow lines to make curves

8. Feature Descriptors for Curves

9. Curve length Curvature Curves smoothness

10. Feature Descriptors for Curves

12. Curve length Curvature Curves smoothness Corners Concave polygons Convex polygons

14. Feature Descriptors for Curves Curve length Curvature Curves smoothness Corners Concave polygons Convex polygons Periodicity Frequency …

15. Feature Descriptors for Regions

16. Matching Regions Two views of the same object Find matching regions

17. Matching Regions

18. Template Matching If regions are the same size we can again treat them as vectors and compare them Sum of Squared Differences (SSD) d = (a 1 -b 1 ) 2 + (a 2 -b 2 ) 2 + … Euclidean Distance without the square root Slightly faster and closer things still have smaller values Normalized Cross Correlation d = a/|a|. b/|b| More invariant to changes in intensity Values between -1 and 1

19. Template Matching: SSD >r0 = mean(imread(‘scar1_region2.png’), 3); >imagesc(r0); >r1 = mean(imread(‘scar2_region1.png’), 3); >r2 = mean(imread(‘scar2_region2.png’), 3); >r3 = mean(imread(‘scar2_region3.png’), 3); >sum((r1(:)-r0(:)).^2) >sum((r2(:)-r0(:)).^2) >sum((r3(:)-r0(:)).^2)

20. Template Matching: NCC >r0 = r0(:) / norm(r0(:)); >r1 = r1(:) / norm(r1(:)); >r2 = r2(:) / norm(r2(:)); >r3 = r3(:) / norm(r3(:)); >dot(r0,r1) >dot(r0,r2) >dot(r0,r3)

21. Color Average Color Treat the entire regions as one color Coarse representation Regions don’t have to be the same size

22. Color >r0 = imread(‘scar1_region2.png’); >image(r0); >whos >mean(mean(r0, 1), 2) >c0 = reshape(mean(r0, 1), 2), 1, 3); >image(uint8(reshape(c0, 1, 1, 3)));

23. Color >r1 = imread(‘scar2_region1.png’); >image(r1); >c1 = reshape(mean(r1, 1), 2), 1, 3); >image(uint8(reshape(c1, 1, 1, 3))); >r2 = imread(‘scar2_region2.png’); >image(r2); >c2 = reshape(mean(r2, 1), 2), 1, 3); >image(uint8(mean(r2, 1), 2))); >r3 = imread(‘scar2_region3.png’); >image(r3); >c3 = reshape(mean(r3, 1), 2), 1, 3); >image(uint8(mean(r3, 1), 2)));

24. Color >sqrt(sum((c1-c0).^2)) >sqrt(sum((c2-c0).^2)) >sqrt(sum((c3-c0).^2))

25. Color Average Color Treats the entire regions as one color Coarse representation Regions don’t have to be the same size Color Distributions Build a distribution of colors in a region A bit more detail Regions still don’t have to be the same size

26. Color >r0 = imread(‘scar1_region2.png’); >r = r0(:,:,1); >g = r0(:,:,2); >b = r0(:,:,3); >hr = hist(double(r(:)), 10); >hg = hist(double(g(:)), 10); >hb = hist(double(b(:)), 10); >h0 = [hr hg hb]; >h0 = h0 / norm(h0);

27. Color >r1 = imread(‘scar2_region1.png’); >… >r2 = imread(‘scar2_region2.png’); >… >r3 = imread(‘scar2_region3.png’); >… >sqrt(sum((h1-h0).^2)) >sqrt(sum((h2-h0).^2)) >sqrt(sum((h3-h0).^2))

28. Color Spaces L  Human studies to model color similarity Euclidean distances between values better matches our perception HSV Hue: The colors Saturation: The purity of the color Value: The brightness, a black and white version of the image Matlab: rgb2hsv, hsv2rgb

29. Texture

30. Break the universe of textures into individual elements of texture, textons Use these textons as patterns to look for in the image and use convolution to find them Gaussian Filters In particular variations of Gaussian filters

31. Texture: Orientations >filter = fspecial(‘guassian’, 50, 5); >filter = diff(filter, 1, 2); >filter = filter / norm(filter); >imagesc(filter); >filter = imrotate(filter, 45); >imagesc(filter);

32. Texture: Elongations >f = fspecial(‘guassian’, 50, 5); >f = diff(f, 1, 2); >f = imresize(f, [2*size(f,1) size(f,2)]); >f = f / norm(f); >imagesc(f);

33. Texture: Miscellaneous >filter = fspecial(‘guassian’, 50, 5); >filter = diff(filter, 1, 2); >filter = filter / norm(filter); >imagesc(filter); >filter = imrotate(filter, 45); >imagesc(filter); >filter = fspeical(‘log’, 50, 5); Bars Spots

34. Texture: Scales >filter = … >filter = imresize(filter, 0.5); >filter = filter / norm(filter); >imagesc(filter);

35. Texture Filter bank Create many filters spanning: Orientations Elongations Types Scales

36. Texture >Ig = mean(imread(‘scar1.jpg’), 3); >Ig = Ig / norm(Ig); >f = fspecial(‘gaussian’, 16, 2); >f1 = diff(f, 1, 2); >f1 = f1 / norm(f1); >f2 = imrotate(f1, 45, ‘bilinear’, ‘crop’); >f2 = f2 / norm(f2); >f3 = imrotate(f2, 45, ‘bilinear’, ‘crop’); >f3 = f3 / norm(f3); >Ir1 = abs(conv2(Ig, f1)); >Ir2 = abs(conv2(Ig, f1)); >Ir3 = abs(conv2(Ig, f1));

37. Texture >Ig = mean(imread(‘textures.png’), 3); >Ig = Ig / norm(Ig); >f = fspecial(‘gaussian’, 16, 2); >f1 = diff(f, 1, 2); >f1 = f1 / norm(f1); >f2 = imrotate(f1, 45, ‘bilinear’, ‘crop’); >f2 = f2 / norm(f2); >f3 = imrotate(f2, 45, ‘bilinear’, ‘crop’); >f3 = f3 / norm(f3); >Ir1 = abs(conv2(Ig, f1)); >Ir2 = abs(conv2(Ig, f1)); >Ir3 = abs(conv2(Ig, f1));

38. Texture >r1 = Ir1(1:100,1:100); >r2 = Ir2(1:100,1:100); >r3 = Ir3(1:100,1:100); >h1 = hist(r1(:), 10); >h2 = hist(r2(:), 10); >h3 = hist(r3(:), 10); >ht1 = [h1 h2 h3]; >ht1 = ht1 / norm(ht1); >r1 = Ir1(1:100,100:200); >… >r1 = Ir1(1:100,200:300); >…

39. Texture >sqrt(sum((ht1-ht2).^2)) >sqrt(sum((ht1-ht3).^2)) >sqrt(sum((ht2-ht3).^2))

40. Miscellaneous Size Count the number of pixels Statistics Calculate mean and standard deviation in x and y directions Neighbors …

41. Image and Spatial Data Analysis Group http://isda.ncsa.illinois.edu Questions?