1. Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Computer Vision Chapter 9 Texture Presented by 盧毅 and 傅楸善教授 Cell phone: 0978868223 E-mail: r02922144@ntu.edu.tw

2. DC & CV Lab. CSIE NTU Copyright Mentioned Some slides adapted from http://courses.cs.washington.edu/courses/cse 455/09wi/Lects/lect12.pdf http://courses.cs.washington.edu/courses/cse 455/09wi/Lects/lect12.pdf Some slides about LBP and variants adapted from XianBiao Qi

3. DC & CV Lab. CSIE NTU Introduction What does texture mean? Formal approach or precise definition of texture does not exist! Texture discrimination techniques are for the part ad hoc.

4. DC & CV Lab. CSIE NTU What is Texture?

5. DC & CV Lab. CSIE NTU “Definition” of Texture Non-local property, characteristic of region larger than its size Repeating patterns of local variations in image intensity which are too fine to be distinguished as separated objects at the observed resolution

6. DC & CV Lab. CSIE NTU “Definition” of Texture (cont.) For humans, texture is the abstraction of certain statistical homogeneities from a portion of the visual field that contains a quantity of information grossly in excess of the observer’s perceptual capacity

7. DC & CV Lab. CSIE NTU

8. DC & CV Lab. CSIE NTU

9. DC & CV Lab. CSIE NTU

10. DC & CV Lab. CSIE NTU

11. DC & CV Lab. CSIE NTU

12. DC & CV Lab. CSIE NTU

13. DC & CV Lab. CSIE NTU

14. DC & CV Lab. CSIE NTU

15. Texture Classification Samples of Brodatz, CUReT, and KTH-TIPS

16. DC & CV Lab. CSIE NTU Texture Analysis Issues Pattern recognition: given texture region, determine the class the region belongs to Generative model: given textured region, determine a description or model for it Texture segmentation: given image with many textured areas, determine boundaries

17. DC & CV Lab. CSIE NTU Statistical Texture-Feature Approaches Autocorrelation function Spectral power density function Edgeness per unit area Spatial gray level co-occurrence probabilities Graylevel run-length distributions Relative extrema distributions Mathematical morphology

18. DC & CV Lab. CSIE NTU Image Texture Analysis Give a generative model and the values of its parameters, one can synthesize homogeneous image texture samples associated with the model and the given value of its parameters.

19. DC & CV Lab. CSIE NTU Image Texture Analysis (cont.) Verification: verify given image textures sample consistent with model Estimation: estimate values of model parameters based on observed sample examples of model-based techniques

20. DC & CV Lab. CSIE NTU Some Model-Based Techniques Autoregressive, moving-average, time-series models (extended to 2D) Markov random fields Mosaic models

21. DC & CV Lab. CSIE NTU Nonfigurative and Cellular Texture It is decomposable It has two basic dimensions First dim: gray level primitives Second dim: spatial organization of the gray level primitives

22. DC & CV Lab. CSIE NTU Nonfigurative and Cellular Texture Gray level primitives: Regions with gray level properties Spatial organization: Random, Pairwise dependence, Dependence of n primitives at a time Dependence: structural, probabilistic, or functional

23. DC & CV Lab. CSIE NTU Texel Texture element The basic textural unit of some textural primitives in their defining spatial relationships A texture is a set of texture elements or texels occurring in some regular or repeated pattern

24. DC & CV Lab. CSIE NTU Texel

25. DC & CV Lab. CSIE NTU Characterizing Texture An image texture is described by the number and types of its primitives and their spatial organization or layout. Image texture can be qualitatively evaluated as some properties.

26. DC & CV Lab. CSIE NTU Some Texture Features Fineness Coarseness Contrast Directionality Roughness Regularity Smoothness Granulation

27. DC & CV Lab. CSIE NTU Some Texture Features (cont.) Randomness Lineation Mottled Irregular Hummocky

28. DC & CV Lab. CSIE NTU Characterizing Texture (cont.) Each of these qualities translates into some property of the gray level primitives and the spatial interaction between them. Open issue: no attempts to map semantic meaning into precise properties of gray level primitives and their spatial distribution.

29. DC & CV Lab. CSIE NTU Aspects of texture

30. DC & CV Lab. CSIE NTU Take a Break

31. DC & CV Lab. CSIE NTU Texture and Scale For any textural surface, there exists a scale at which, when the surface is examined, it appears smooth and textureless. (see from infinite distance) As resolution increases, the surfaces appears as a fine texture and then a coarse one, and for multiple-scale textural surface the cycle of smooth, fine, and coarse may repeat.

32. DC & CV Lab. CSIE NTU Texture and Scale (cont.) Thus, texture cannot be analyzed without frame of reference on scale or resolution. Texture is a scale-dependent phenomenon.

33. DC & CV Lab. CSIE NTU

34. DC & CV Lab. CSIE NTU First-Order Gray-Level Statistics Statistics of single pixels E.g. Histogram, mean, median, variance

35. DC & CV Lab. CSIE NTU Second-Order Gray-Level Statistics The combined statistics of gray levels of pairs of pixels in which each two pixels in a pair have a fixed relative position E.g. co-occurrence Gray level spatial dependence: characterize texture by co-occurrence

36. DC & CV Lab. CSIE NTU Co-Occurrence Matrix The gray level co-occurrence can be specified in a matrix of relative frequencies P ij with which two neighboring pixels separated by distance d occur on the image, one with gray level i and the other with gray level j Symmetric matrix Function of angle and distance between pixels

37. DC & CV Lab. CSIE NTU 2)

38. DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Probability of horizontal, d pixels apart pixels P(i, j, d, 0°) = #{[(k, l), (m, n)] | k-m = 0, |l-n| = d, I(k, l) = i, I(m,n) = j} Probability of 45 °, d pixels apart pixels P(i, j, d, 45°) = #{[(k, l), (m, n)] | (k-m = d, l-n = -d) or (k-m = -d, l-n = d), I(k, l) = i, I(m,n) = j}

39. DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Probability of 90 °, d pixels apart pixels P(i, j, d, 90°) = #{[(k, l), (m, n)] | |k-m| = d, l-n = 0, I(k, l) = i, I(m,n) = j} Probability of 135 °, d pixels apart pixels P(i, j, d, 135°) = #{[(k, l), (m, n)] | (k-m = d, l-n = d) or (k-m = -d, l-n = -d), I(k, l) = i, I(m,n) = j}

40. DC & CV Lab. CSIE NTU 0

41. DC & CV Lab. CSIE NTU Co-Occurrence Matrix (cont.) Matrix symmetric: P(i, j, d, a) = P(j, i, d, a)

42. DC & CV Lab. CSIE NTU Variants of co-occurrence matrix Gray level difference probability: The probability of small contrast d for a coarse texture will be much higher than for a fine texture.

43. DC & CV Lab. CSIE NTU Variants of co-occurrence matrix (cont.) Another gray level difference probability: Can be used to compute a variety of features, such as entropy and energy.

44. DC & CV Lab. CSIE NTU Take a Break

45. DC & CV Lab. CSIE NTU

46. DC & CV Lab. CSIE NTU Matrix with Highest Entropy When all entries in P ij are equal Image where no preferred gray-level pairs exist features calculated from the co- occurrence matrix

47. DC & CV Lab. CSIE NTU Generalized Gray Level Spatial Dependence Models for Texture Simple generalization: consider more than two pixels at a time

48. DC & CV Lab. CSIE NTU Generalized Co-Occurrence Primitives and the spatial relationships between primitives. Difference from gray level co-occurrence: Pixel vs. Primitives Strong texture measures take into account the co-occurrence between texture primitives.

49. DC & CV Lab. CSIE NTU Texture Primitive Connected set of pixels characterized by attribute set Simplest primitive: pixel with gray level attribute More complicated primitive: connected set of pixels homogeneous in level, characterized by size, elongation, orientation, and average gray level

50. DC & CV Lab. CSIE NTU Spatial Relationship We have a list of primitives, their center coordinate, and their attributes after the primitives have been constructed.

51. DC & CV Lab. CSIE NTU Spatial Relationship (cont.)

52. DC & CV Lab. CSIE NTU Take a Break

53. DC & CV Lab. CSIE NTU Texture feature State-of-the-art works: "Pairwise rotation invariant co-occurrence local binary pattern“ ECCV 2012 "Multi-scale Joint Encoding of Local Binary Patterns for Texture and Material Classification“ BMVC 2013

54. DC & CV Lab. CSIE NTU Local Binary Pattern Classical image feature Successfully used in many CV applications Firstly proposed in > TPAMI 2002 > TPAMI 2006

55. Local Binary Pattern 564657 877488 123117121 000 11 111 00011111 LBP 8,1

56. Local Binary Pattern Formula:

57. Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation Divide LBP patterns into several subgroups 01111111

58. Rotation Invariant LBP Pattern LBP describes image microstructures There are 256 LBP 8,1 patterns Rotationpermutation LBP 8,1 pattern has 36 subgroups under rotation permutations

59. Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Samples of non-uniform patterns Samples of uniform patterns

60. Uniform LBP Pattern At most two transitions from 0 1 Most patterns are uniform pattern in images Sometimes >90% Microstructures defined by uniformed patterns Bright spot (0) Flat area or dark spot (8) Edges of varying positive/negative curvature (1-7)

61. DC & CV Lab. CSIE NTU Take a Break

62. DC & CV Lab. CSIE NTU State-of-the-art works: "Pairwise rotation invariant co-occurrence local binary pattern“ ECCV 2012 "Multi-scale Joint Encoding of Local Binary Patterns for Texture and Material Classification“ BMVC 2013

63. Multi-Scale Joint encoding Local Binary Patterns (MSJ-LBP)

64. LBP and Multi-scale LBP The LBP depicts the local structures of the images such as plat regions, edge, contours and so on. Contrast to LBP, multi-scale LBP describes larger and stronger structures, as shown in Fig. 1. Meanwhile, in the real image, the LBP patterns in different scales has strong relationship as shown in the right of Fig.1. LBPLBP on multiple scales To illustrate the LBP and Multi-scale LBP. MS-LBP depicts stronger structures than LBP. 1 0 A A

65. LBP and Multi-scale LBP

66. Dense Sampling 1 0

67. To illustrate the rotation invariant property of the proposed MSJ-LBP. Rotation Invariant Property Fig. 2 illustrates the rotation invariant property of the proposed MSJ-LBP. As shown in Fig. 2, the left MSJ-LBP pattern is [(11110000 )ru, (11110001)u]. Similarly, we can obtain the same pattern from the right of Fig. 2.

68. Drawback of MS-LBP

69. Experiment

70. Experiment (cont.)

71. DC & CV Lab. CSIE NTU Take a Break

72. Pairwise Rotation Invariant Co- occurrence Local Binary Pattern

73. Co-LBP pattern space is large 2 16 Co-occurrence Local Binary Pattern 1 1 0 0 0 1 1 1 A 0 0 0 1 1 1 1 1 B CoLBP(AB) ： 01111100 00011111

74. Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Co-occurrence Local Binary Pattern AB CoLBP(AB) ： 0111110000011111 A* B* CoLBP(A * B * ) ： 0001111111000111 90 o

75. Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Pairwise rotation invariant CoLBP Co-occurrence Local Binary Pattern AB CoLBP pri (AB) ： 00011111 11000111 CoLBP ri (A) ： 00011111

76. Co-LBP pattern space is large 2 16 Co-LBP under rotation permutation. Pairwise rotation invariant CoLBP Uniformed pairwise–rotation invariant COLBP 10*59 patterns Pairwise Rotation Invariant Co- occurrence Local Binary Pattern

77. The Relative Angle Information The relative angle information is informative and rotation invariant.

78. Rotation Invariant Pairwise Rotation Invariant encoding just promises that for the same A and B, the encoding strategy is rotation invariant. But we must keep that for same point A in different images, the point B could be found. The gradient orientation is a relative invariant between point A and B in different images.

79. The difference between the invariant properties of co-occurrence feature and traditional features Same pairs should be promised. Invariant encoding strategy should be used

80. Encoding Multi-Scale and Multi- Orientation Information AB 2 46 C D E F G CoLBP(A B) CoLBP(A C) CoLBP(A D) CoLBP(A E) CoLBP(A F) CoLBP(A G) Vector dimension : 590*6 MSMO-CoLBP: MSMO- CoLBP R MSMO- CoLBP G MSMO- CoLBP B Vector dimension : 590*6*3 Color MSMO-CoLBP Vector dimension : 150 PCA dimension Reduction across images Encoding color information

81. DC & CV Lab. CSIE NTU Autocorrelation Function Texture relates to the spatial size of the gray level primitives on an image Gray level primitives of larger size are indicative of coarser texture Gray level primitives of smaller size are indicative of finer texture

82. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) Autocorrelation function describes the size of gray level primitives

83. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

84. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

85. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

86. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.)

87. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) - 1D Example from Wiki

88. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) - Interpreting autocorrelation

89. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) If the gray level on image is relatively large: texture is coarse, autocorrelation drops off slowly with distance If the gray level on image is relatively small: texture is fine, autocorrelation drops off quickly with distance Periodic

90. DC & CV Lab. CSIE NTU Autocorrelation Function (cont.) The gray level primitive in the autocorrelation model is the gray level. The spatial organization is characterized by the correlation coefficient that is a measure of the linear dependence one pixel has on another. The relationship between the autocorrelation function and the power spectral density function: Fourier transforms

91. DC & CV Lab. CSIE NTU Take a Break

92. DC & CV Lab. CSIE NTU Digital Transform Methods and Texture In the digital transform method of texture analysis, the digital image is typically divided into a set of non-overlapping small square subimages. The vectors is rexpressed in a new coordinate system. Fourier transform uses the complex sinusoid basic set, Handamard transfer uses the Walsh function basic set, …..

93. DC & CV Lab. CSIE NTU Digital Transform Methods and Texture (cont.) The key point: The basis vectors of the new coordinate system have an interpretation that relates to spatial frequency or sequency. Since frequency is a close relative of texture, such transformations can be useful.

94. DC & CV Lab. CSIE NTU Digital Transform Methods and Texture (cont.) In general, features based on Fourier power spectra have been shown to perform more poorly than features based on second-order gray level co-occurrence statistics or those based on first-order statistics of spatial gray level differences.

95. DC & CV Lab. CSIE NTU Take a Break

96. DC & CV Lab. CSIE NTU Texture Energy The image is first convolved with a variety of kernels Then each convolved image is processed with a nonlinear operator to determine the total textural energy in each pixel’s neighborhood

97. DC & CV Lab. CSIE NTU Texture Energy (cont.) The textural energy approach is very much in the spirit of the transform approach. But it uses smaller windows or neighborhood supports.

98. DC & CV Lab. CSIE NTU Texture Energy (cont.)

99. DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

100. DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

101. DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

102. DC & CV Lab. CSIE NTU Texture Energy (cont.) - Law’s texture

103. DC & CV Lab. CSIE NTU Texture Energy (cont.) - Using texture energy for segmentation

104. DC & CV Lab. CSIE NTU Take a Break

105. DC & CV Lab. CSIE NTU Texture Edgeness Autocorrelation function and digital transform both reference texture to spatial frequency Texture Edgeness: conceive texture in terms of edgeness per unit area

106. DC & CV Lab. CSIE NTU Texture Edgeness (cont.) Use small neighborhood to detect microedge Use large neighborhood to detect macroedge

107. DC & CV Lab. CSIE NTU

108. DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

109. DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

110. DC & CV Lab. CSIE NTU Texture Edgeness (cont.)

111. DC & CV Lab. CSIE NTU Conclusion of Statistical Approach Co-occurrence probability Autocorrelation Function Texture Energy Texture Edgeness

112. DC & CV Lab. CSIE NTU Conclusion of Statistical Approach

113. DC & CV Lab. CSIE NTU Take a Break