1. Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Computer and Robot Vision I Chapter 3 Binary Machine Vision: Region Analysis Presented by: 傅楸善 & 盧毅 0978868223 r02922144@ntu.edu.tw 指導教授 : 傅楸善 博士

2. DC & CV Lab. CSIE NTU 3.0 Outline Part 1 (textbook content): 1. Region properties 2. Signature segmentation properties Part 2 (Advanced topic on image feature): 1. LBP (Local Binary Pattern) 2. CoLIOP (Co-occurrence Local Intensity Order Pattern)

3. DC & CV Lab. CSIE NTU 3.2 Map of Region Properties 1. Basic properties 2. Extremal Points 3. Spatial Moments 4. Mixed Spatial Gray Level Moments

4. DC & CV Lab. CSIE NTU 3.2 Map of Region Properties

5. DC & CV Lab. CSIE NTU 3.2 Map of Region Properties 1. Basic properties Microtexture properties ( Co-occurrence) 1) texture second moment 2) texture entropy 3) texture contrast 4) texture homogeneity 5) texture correlation

6. DC & CV Lab. CSIE NTU 3.2 Region Properties bounding rectangle: smallest rectangle circumscribes the region area: centroid: 1 1 1 1 111 1 1 1 1 1111 111100 0 0 0000 0 0 0 0 0 0 0 123456 1 2 3 4 5 6 00000 0 0 0 0 0 0 0 00 00000000 0 0 0 0 0 0 7 7 0 0 1 1 A=21 r=3.476 c=4.095

7. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) border pixel: has some neighboring pixel outside the region : 4-connected perimeter: if 8-connectivity for inside and outside : 8-connected perimeter: if 4-connectivity for inside and outside

8. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) N8(r,c) R ( 1, 0 )

9. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) N8(r,c) R ( 1, 1 )

10. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) N4(r,c) R ( 1, 0 )

11. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) N4(r,c) R ( 1, 1 )

12. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) Eg: center is in but not in for

13. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) length of perimeter, successive pixels neighbors where k+1 is computed modulo K i.e.

14. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) P4

15. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) P8

16. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) where k+1 is computed modulo K length of perimeter |P 8 | K = 0,1,2,3,

17. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) mean distance R from the centroid to the shape boundary standard deviation R of distances from centroid to boundary

18. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

19. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

20. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

21. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

22. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) Haralick shows that has properties: 1. digital shape circular, increases monotonically 2. similar for similar digital/continuous shapes 3. orientation (rotation) and area (scale) independent

23. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) Average gray level (intensity) Gray level (intensity) variance

24. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) microtexture properties: A function of co- occurrence matrix S: set of pixels in designated spatial relationship e.g. 4-neighbors Define the region’s co-occurrence matrix P by

25. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

26. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’)

27. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) 0 0 1 2 3 01230123

28. DC & CV Lab. CSIE NTU

29. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) texture second moment (Haralick, Shanmugam, and Dinstein, 1973)

30. texture entropy DC & CV Lab. CSIE NTU

31. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) texture contrast

32. DC & CV Lab. CSIE NTU 3.2 Region Properties (cont’) texture homogeneity where k is some small constant

33. texture correlation where DC & CV Lab. CSIE NTU

34. DC & CV Lab. CSIE NTU 3.2 Map of Region Properties 2. Extremal Points Definition 1) points 2) axes 3) length 4) orientation Three cases 1) linelike shape 2) Triangular shape 3) square and rectangular shape 4) octagonal shape

35. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points eight distinct extremal pixels: topmost left, topmost right, rightmost top, rightmost bottom, bottommost right, bottommost left, leftmost bottom, leftmost top,

36. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’)

37. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) different extremal points may be coincident

38. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) association of the name of the eight extremal points with their coordinates

39. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) association of the name of an external coordinate with its definition

40. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) directly define the coordinates of the extremal points:

41. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) extremal points occur in opposite pairs: topmost left bottommost right, topmost right bottommost left, rightmost top leftmost bottom, rightmost bottom leftmost top each opposite extremal point pair: defines an axis axis properties: length, orientation

42. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) the length covered by two pixels horizontally adjacent 1: distance between pixel centers 2: from left edge of left pixel to right edge of right pixel

43. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) distance calculation: add a small increment to the Euclidean distance

44. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) length going from left edge of left pixel to right edge of right pixel θ x

45. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) orientation taken counterclockwise w.r.t. column (horizontal) axis

46. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) orientation for the axes axes paired: with and with

47. DC & CV Lab. CSIE NTU r c α

48. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) calculation of the axis length and orientation of a linelike shape

49. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) distance between ith and jth extremal point average value of = 1.12, largest error 0.294 = - 1.12

50. DC & CV Lab. CSIE NTU calculations for length of sides base and altitude for a triangle

51. DC & CV Lab. CSIE NTU calculation for the orientation of an example rectangle

52. DC & CV Lab. CSIE NTU calculation for the orientation of an example rectangle

53. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’) axes and their mates that arise from octagonal-shaped regions

54. DC & CV Lab. CSIE NTU 3.2.1 Extremal Points (cont’)

55. DC & CV Lab. CSIE NTU 3.2 Map of Region Properties 3. Spatial Moments 1) Second-order row moment 2) Second-order mixed moment 3) Second-order column moment 4) Second-order mixed gray level spatial moment

56. DC & CV Lab. CSIE NTU 3.2.2 Spatial Moments( 矩 ) Second-order row moment Second-order mixed moment Second-order column moment

57. If a region R is an ellipse whose center is the origin A relationship given by DC & CV Lab. CSIE NTU

58. DC & CV Lab. CSIE NTU 3.2.3 Mixed Spatial Gray Level Moments (cont’) region properties: position, shape, gray level properties Second-order mixed gray level spatial moments

59. 3.2.3 Mixed Spatial Gray Level Moments The mixed gray level spatial moments can be used to determine the least-squares gray level intensity planes to the observed gray level spatial pattern of the region R DC & CV Lab. CSIE NTU

60. DC & CV Lab. CSIE NTU 3.2.3 Mixed Spatial Gray Level Moments (cont’) connected components labeling of the image in Fig 2.2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 01 02 03 04 05 06 07 08 09 10 11 12 13

61. DC & CV Lab. CSIE NTU all the properties measured from each of the regions 3.2.3 Mixed Spatial Gray Level Moments (cont’)

62. DC & CV Lab. CSIE NTU 3.2.3 Mixed Spatial Gray Level Moments (cont’)

63. DC & CV Lab. CSIE NTU 3.3 Signature Properties Signature or Projection Area Centroid Second moment Bounding rectangle Rectangle: orientation & position Circle: position

64. DC & CV Lab. CSIE NTU 3.3 Signature Properties vertical projection horizontal projection diagonal projection from lower left to upper right diagonal projection from upper left to lower right

65. DC & CV Lab. CSIE NTU 3.3 Signature Properties

66. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) Compute properties from projections Area, centroid of the region, second moments area

67. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) rmin: top row of bounding rectangle rmax; bottom row of bounding rectangle cmin: leftmost column of bounding rectangle cmax: rightmost column of bounding rectangle

68. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) row centroid column centroid diagonal centroid another diagonal centroid

69. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) second row moment from horizontal projection second column moment from vertical projection second diagonal moment

70. DC & CV Lab. CSIE NTU 3.3 Signature Properties

71. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) second diagonal moment related to second mixed moment can be obtained from projection second diagonal moment related to

72. DC & CV Lab. CSIE NTU 3.3 Signature Properties (cont’) second mixed moment can be obtained from projection mixed moment obtained directly from and

73. DC & CV Lab. CSIE NTU 3.3.1 Signature Analysis to Determine the Center and Orientation of a Rectangle signature analysis: important because of easy, fast implementation surface mount device (SMD) inspection: position and orientation of parts

74. DC & CV Lab. CSIE NTU 3.3.1 Signature Analysis to Determine the Center and Orientation of a Rectangle (cont’) determine center of rectangle by corner location side lengths w, h orientation angle

75. DC & CV Lab. CSIE NTU geometry for determining the translation of the center of a rectangle h/2 -w/2

76. DC & CV Lab. CSIE NTU partition rectangle into six regions formed by two vertical lines a known distance g apart and one horizontal line

77. DC & CV Lab. CSIE NTU 3.3.1 Signature Analysis to Determine the Center and Orientation of a Rectangle (cont’)

78. DC & CV Lab. CSIE NTU

79. DC & CV Lab. CSIE NTU where rotation angle 3.3.1 Signature Analysis to Determine the Center and Orientation of a Rectangle (cont’)

80. DC & CV Lab. CSIE NTU 3.3.2 Using Signature to Determine the Center of a Circle (cont’) circle projected onto the four quadrants of the projection index image

81. DC & CV Lab. CSIE NTU

82. DC & CV Lab. CSIE NTU

83. DC & CV Lab. CSIE NTU 3.3.2 Using Signature to Determine the Center of a Circle (cont’) each quadrant area from histogram of the masked projection positive if A + B > C + D negative otherwise where positive if B + D > A + C, negative otherwise

84. DC & CV Lab. CSIE NTU 3.4 Summary region properties from connected components or signature analysis

85. DC & CV Lab. CSIE NTU Histogram Equalization (Homework) a method in image processing of contrast adjustment using the image's histogram pixel transformation r, s: original, new intensity, T: transformation T( r ) single-valued, monotonically increasing for

86. DC & CV Lab. CSIE NTU Part 2: Image Features The traditional pattern recognition and computer vision task preprocessing feature extractionclassifier Datainformation

87. DC & CV Lab. CSIE NTU Part 2: Image Features Texture Classification

88. DC & CV Lab. CSIE NTU Part 2: Image Features Flower Recognition

89. DC & CV Lab. CSIE NTU Part 2: Image Features Face Recognition

90. DC & CV Lab. CSIE NTU Part 2: Image Features LBP (Local Binary Pattern) CLIOP ( Co-occurrence of Local Intensity Order )

91. DC & CV Lab. CSIE NTU Part 2: Local Binary Pattern Classical image feature Successfully used in many CV applications Firstly proposed in > TPAMI 2002 > TPAMI 2006

92. Local Binary Pattern 564657 877488 123117121 000 11 111 00011111 LBP 8,1

93. Local Binary Pattern Formula:

94. Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation Divide LBP patterns into several subgroups 01111111

95. Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation LBP 8,1 pattern has 36 subgroups under rotation permutations

96. Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Samples of non-uniform patterns Samples of uniform patterns

97. Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Microstructures defined by uniformed patterns Bright spot (0) Flat area or dark spot (8) Edges of varying positive/negative curvature (1-7)

98. Part2: CoLIO for face recognition Feature is key for face recognition tasks Robustness to variances and high discriminative power Inspired by: 1. Coding scheme: local intensity order pattern 2. Enhance strategy: Co-occurrence

99. Coding scheme of CoLIO LIO: CoLIO:

100. Insight of Co-occurrence The num of all possible permutations of >4 pixels is little high Spatial information for face representation

101. Framework of CoLIO

102. Evaluation of CoLIO configurations AR Dataset The results show the power of the co-occurrence that can catch informative spatial relationships.

103. Evaluation of CoLIO configurations LFW Dataset

104. Experiments of CoLIO on AR dataset Experiment 1: The results show the CoLIO is more powerful than LBP in face recognition task. Experiment 2:

105. Experiments of CoLIO on UMIST Experiment 3:

106. Experiments of CoLIO on LFW Exp 4:

107. Conclusion of CoLIO Local intensity order is firstly applied for face recognition Simplicity Extremely fast extraction speed robust to illumination and preserve discriminative power “Spatial Co-occurrence of Local Intensity Order for Face Recognition”, ICME 2013 Workshop （ More details can be found ）

108. DC & CV Lab. CSIE NTU

109. DC & CV Lab. CSIE NTU histogram equalization histogram linearization number of pixels with intensity j n: total number of pixels for every pixel if then Histogram Equalization (Homework)

110. DC & CV Lab. CSIE NTU Project due Oct. 9 Write a program to do histogram equalization Histogram Equalization (Homework)

111. DC & CV Lab. CSIE NTU CVers 畢生追求的是漫畫家手中的不變量

112. DC & CV Lab. CSIE NTU End