1. Chapter 8 Image Representation & Analysis
Chuan-Yu Chang (張傳育)Ph.D.
Dept. of Computer and Communication Engineering
National Yunlin University of Science & Technology
Office: EB212
Tel: Ext. 4337

2. Image Representation
To perform a computerized analysis of an image, it is important to establish a hierarchical framework of processing steps representing the image and knowledge domain.
The bottom-up analysis starts with the analysis at the pixels-level representation and moves up toward the understanding of the scene or the scenario.
The top-down analysis starts with the hypothesis for the presence of an object and then moves toward the pixel-level representation to verify or reject the hypothesis using the knowledge-based models.

3. Image Representation Top-Down Bottom-Up Scenario Scene-1 Scene-I
Object-1
Object-J
S-Region-1
S-Region-K
Region-1
Region-L
Pixel (i,j)
Edge-M
Edge-1
Pixel (k,l)
Top-Down

4. Image Representation
Knowledge-based models can be used at different stages of processing.
The knowledge of physical constraints and tissue properties can be very useful in imaging and image reconstruction.
Anatomical locations of various organs in the body often impose a challenge in imaging the desired tissue or part of the organ.
An object representation model usually provides the knowledge about the shape or other characteristic features of a single objects for the classification analysis.

5. Knowledge Domain Data Domain
Image Reconstruction
Image
Segmentation
(Edge and Region)
Feature Extraction
and
Representation
Classification
Object Identification
Analysis
of Classified Objects
Multi-Modality/Multi-Subject/Multi-Dimensional Registration, Visualization and Analysis
Raw Data from Imaging System
Single Image
Understanding
Multi-Modality/ Multi-Subject/Multi-Dimensional
Image Understanding
Scene
Models
Object
Feature
Edge/Region Representation
Physical Property/
Constraint
Knowledge Domain
Data
Domain

6. Feature Extraction
After segmentation, specific features representing the characteristics and properties of the segmented regions in the image need to be computed for object classification and understanding.
There are four major categories of features for region representation:
Statistical Features
Provide quantitative information about the pixels within a segmented region.
Ex: Histogram, Moments, Energy, Entropy, Contrast, Edges

7. Image Analysis: Feature Extraction
Shape Features
Provide information about the characteristic shape of the region boundary.
Ex: Boundary encoding, Moments, Hough Transform, Region Representation, Morphological Features
Texture Features
Provide information about the local texture within the region or the corresponding part of the image.
Ex: second-order histogram statistics, co-occurrence matrix, wavelet processing.
Relational Features
Provide information about the relational and hierarchical structure of the regions associated with a single or a group of objects.

8. Statistical Pixel-Level Features
The histogram of the gray values of pixels
Mean of the gray values of the pixels
Variance and central moments in the region where n=2 is the variance of the region. n=3 is a measure of noncentrality n=4 is a measure of flatness of the histogram.

9. Statistical Pixel-Level Features
Energy: Total energy of the gray-values of pixels
Entropy 熵
Local contrast
Maximum and minimum gray values

10. Shape Features
The shape of a region is defined by the spatial distribution of boundary pixels.
Circularity, compactness, and elongatedness through the minimum bounded rectangle that covers the region.
Several features using the boundary pixels of the segmented region can be computed as
Chain code for boundary contour
Fourier descriptor of boundary contour
Central moments based shape features for segmented region
Morphological shape descriptors

11. Some Shape Features Longest axis GE. Shortest axis HF.
Perimeter and area of the minimum bounded rectangle ABCD.
Elongation ratio: GE/HF
Perimeter p and area A of the segmented region.
Circularity
Compactness A E H D B C F G O

12. Boundary Encoding :Chain Code
Define a neighborhood matrix with the orientation primitives with respect to the center pixel.
The code of specific orientation are set for 8-connected neighborhood directions.
The orientation directions are codes with a numerical value ranging from 0 to 7.
The boundary contour needs to be approximated as a list of segments that have pre-selected length and directions.

13. Boundary Encoding :Chain Code
To obtain boundary segments representing a piecewise approximation of the original boundary contour, the “divide and conquer ” is applied.
Selects two points on a boundary contour as vertices.
A straight line joining the two selected vertices can be used to approximate the respective curve segment if it satisfies a “maximum-deviation” criterion for no further division of the curve segment.
The maximum deviation criterion is based on the perpendicular distance between any point on the original curve segment between the selected vertices and corresponding approximated straight-line segment.

14. Boundary Encoding :Chain Code
If the perpendicular distance or deviation of any point on the curve segment from the approximated straight-line segment exceeds a pre-selected deviation threashold, the curve segment is further divided at the point of maximum deviation.
This process of dividing the segments with additional vertices continues until all approximated straight-line segments satisfy the maximum-deviation criterion.
The representation is further approximated using the orientation primitive of the 8-connected neighborhood.
Two parameters can change the chain code: number of orientation primitives and the maximum deviation threshold used in approximating the curve.

15. Boundary Encoding :Chain Code4 2 3 1 5 6 7 4 2 3 1 5 6 7 xc
The 8-connected neighborhood codes (left) and the orientation directions (right) with respect to the center pixel xc.

16. 選取在方向及梯度上有較明顯的兩的頂點為起始點F A D C E B
Chain Code:
BF大於預設值，
需將AC分成AB, BC
A schematic example of developing chain code for a region with boundary contour ABCDE. From top left to bottom right: the original boundary contour, two points A and C with maximum vertical distance parameter BF, two segments AB and BC approximating the contour ABC, five segments approximating the entire contour ABCDE, contour approximation represented in terms of orientation primitives, and the respective chain code of the boundary contour.

17. Boundary Encoding: Fourier Descriptor
Fourier series may be used to approximate a closed boundary of a region.
Assume that the boundary of an object is expressed as a sequence of N points with the coordinates u[n]={x(n), y(n)}, such that
The discrete Fourier Transform of the sequence u[n] is the Fourier descriptor Fd[n] of the boundary and is defined as

18. Boundary Encoding: Fourier Descriptor
Rigid geometric transformation of a boundary such as translation, rotation and scaling can be represented by simple operations on its Fourier transform.
The Fourier descriptors can be used as shape descriptors for region matching dealing with translation, rotation and scaling.

19. Moments for Shape Description
The shape of a boundary or contour can be represented quantitatively by the central moments for matching.
The central moments represent specific geometrical properties of the shape and are invariant to the translation, rotation and scaling.
The central moments mpqof a segmented region or binary image f(x,y) are given by

20. Moments for Shape Description
The normalized central moments are defined as
There are seven invariant moments for shape matching

21. Morphological Processing for Shape Description
Mathematical morphology
A tool for extracting image components that are useful in the representation and description of region shape, such as boundaries, skeletons, and convex hull.
Sets in mathematical morphology represent objects in an image.
2D integer space Z2
(x,y) coordinates
Z3: gray-scale digital images
(x,y) coordinates, and gray-level value

22. Morphological Processing for Shape Description (cont.)
Let A be a set in Z2, If a=(a1, a2) is an element of A
If a is not an element of A, we write
The set with no elements is called the null or empty set and denoted by the symbol
The elements of the sets with which we are concerned are the coordinates of pixels representing objects.
Ex: set C is the set of elements, w, such that w is formed by multiplying each of the two coordinates of all the elements of set D by -1.
(9.1-1)
(9.1-2)

23. Morphological Processing for Shape Description (cont.)
Basic Concepts from Set Theory
Subset
If every element of a set A is also an element of another set B, then A is said to be a subset of B.
Union
The set of all elements belonging to either A, B, or both
Intersection
The set of all elements belonging to both A and B
(9.1-3)
(9.1-4)
(9.1-5)

24. Morphological Processing for Shape Description (cont.)
Disjoint (mutually excusive)
If the two set have no common elements
Complement:
The complement of a set A is the set of elements not contained in A
Difference:
the set of elements that belong to A, but not to B.
(9.1-6)
(9.1-7)
(9.1-8)

25. Morphological Processing for Shape Description (cont.)

26. Preliminaries (cont.)
Reflection
Translation
(9.1-9)
(9.1-10)

27. Logic Operations Involving Binary Images
The principal logic operations used in image processing are AND, OR, and NOT
The three basic logical operations
Performed on a pixel by pixel basis between corresponding pixels of two or more images.
Logical operation are restricted to binary variables
These operations are functionally complete in the sense that they can be combined to form any other logic operation

28. Logic Operations Involving Binary Images
Black indicates a binary 1
White indicates a 0.

29. Dilation and Erosion For sets A and B in Z2
The dilation of A by B, denoted where set B is referred to as the structuring element.
The dilation of A by B is the set of all displacements, z, such that and A overlap by at least one element.
(9.2-1)
(9.2-2)

30. Dilation and Erosion (cont.)

31. Morphological Processing for Shape Description
Set A
Set B
A large region with square shape representing the set A and a small region
with rectangular shape representing the structuring element set B.

32. : Dilation of A by B
: Erosion of A by B A
The dilation of set A by the structuring element set B (top left), the erosion of set A by the structuring element set B (top right) and the result of two successive erosions of set A by the structuring element set B (bottom).

33. Dilation and Erosion (cont.)
Example of dilation
bridging gaps
The maximum length of the breaks is known to be two pixels.
A simple structuring element that can be used for repairing the gaps is shown in Fig. 9.5(b)

34. Dilation and Erosion For sets A and B in Z2
The erosion of A by B, denoted where set B is referred to as the structuring element.
The erosion of A by B is the set of all points z such that B, translated by z, is contained in A.

35. Dilation and Erosion

36. Morphological FeaturesB

37. Dilation and Erosion Example of erosion -eliminating irrelevant detail
使用13x13的方形結構，對圖(a)進行erosion
使用13x13的方形結構，對圖(b)進行dilation
Example of erosion
-eliminating irrelevant detail

38. Opening and Closing Opening
Generally smoothes the contour of an object, breaks narrow isthmuses, and eliminates thin protrusions.
The opening A by B is the erosion of A by B, followed by a dilation of the result by B.
View the structuring element B as a flat “rolling ball”
The boundary of is then established by the points in B that reach the farthest into the boundary of A as B is rolled around the inside of this boundary.

39. Opening and Closing Closing
Tends to smooth sections of contours, fuses narrow breaks and long thin gulfs, eliminates small holes, and fills gaps in the contour.
The closing of set A by structuring element B, denoted
The closing of A by B is simply the dilation of A by B, followed by the erosion of the result by B.

40. Opening and Closing

41. Opening and Closing
The opening operation satisfies the following properties
AB is a subset of A
If C is a subset of D, then C  B is a subset of D °B
(A  B)  B=A  B

42. Opening and Closing The properties of closing operation
A is a subset of AB
If C is a subset of D, then C  B is a subset of D  B
(A  B)  B=A  B
Multiple openings or closings of a set have no effect after the operator has been applied once.

43. Opening and Closing

44. Morphological Processing for Shape DescriptionB
The morphological opening and closing of set A (top left) by the structuring element set B (top right): opening of A by B (bottom left) and closing of A by B (bottom right).

46. Example of morphological operations on MR

47. Texture Features Texture is an important spatial property .
There are three major approaches to represent texture
Statistical
Based on region histograms, their extensions and their moments.
Representing the high-order distribution of gray values in the image are used for texture representing.
Structural
Arrangements of pre-specified primitives in texture representation, such as a repetitive arrangement of square and triangular shapes.
Spectral
Based on the autocorrelation function of a region or on the power distribution in Fourier transform domain.
Texture is represented by a group of specific spatio-frequency components, such as Fourier and wavelet transform.

48. Texture Features Gray-level co-occurrence matrix (GLCM)
Exploits the high-order distribution of gray values of pixels that are defined with a specific distance or neighborhood criterion.
GLCM P(i,j) is the distribution of the number of occurrences of a pair of gray values i and j separated by a distance vector d=[dx, dy]
The GLCM can be normalized by dividing each value in the matrix by the total number of occurences providing the probability of occurrence of a pair of gray values separated by a distance vector.
Statistical texture features are computed from the normalized GLCM.
The second-order histogram H(yq, yr, d) representing the probability of occurrence of a pair of gray values yq and yr separated by a distance vector d.

49. Gray Level Co-occurrence Matrix (GLCM)1 2 3 0o
45o
90o
135o
The four direction for the GLCM
Gray
Level 1 2 3
Co-occurrence matrix for 45o

50. Gray Level Co-occurrence matrix (GLCM)
Figure (a) A matrix representation of a
5x5 pixel image with three gray values; (b) the GLCM P(i,j) for d=[1,1].

51. Texture Feature Entropy of H(yq, yr, d)
The entropy is a measure of texture nonuniformity
Angular Second Moment of H(yq, yr, d)
ASMH indicates the degree of homogeneity among textures
Contrast of H(yq, yr, d)
(yq, yr) is a measure of intensity similarity

52. Texture Feature Inverse Difference Moment of H(yq, yr, d), IDMH
Provides a measure of the local homogeneity among texture
Correlation of H(yq, yr, d)
The correlation attribute is large for similar elements of the second-order histogram.

53. Texture Feature Mean of H(yq, yr, d), mHm Deviation of Hm(yq, d), dHm
The mean characterizes the nature of the gray-level distribution
Deviation of Hm(yq, d), dHm
Indicates the amount of spread around the mean of the marginal distribution.
Entropy of Hd(ys, d), SHd(ys,d)

54. Texture Feature Angular Second Moment of Hd(ys, d), ASM Hd(ys, d)
Mean of Hd(ys, d), mHd(ys, d),

55. malignant lesion of X-ray mammogram Benign lesion of X-ray mammogram
GLCM of Fig. (a)
GLCM of Fig. (b)

56. Relational Features Relational features
Provide information about adjacencies, repetitive patterns and geometrical relationships among regions of an object.
Could be extended to describe the geometrical relationships among objects in an image or a scene.
The relational features can be described in the form of graphs or rules using a specific syntax or language
The quad-tree based region descriptors can be used for object recognition and classification using the tree matching algorithms.

57. Relational Features
Figure 8.13: A block representation of an image with major
quad partitions (top) and its quad-tree representation.

58. Relational Features
A tree structure representation of brain ventricles
for applications in brain image segmentation and analysis A C B D F I E B C A I E D F

59. Feature and Image Classification
Features selected for image representation are classified for object recognition and characterization
Feature Based Pattern Classifiers
Statistical Pattern Recognition
Unsupervised Learning
Supervised Learning
Syntactical Pattern Recognition
Logical predicates
Rule-Based Classifiers
Model-Based Classifiers
Artificial Neural Networks

60. Feature and Image Classification
Statistical Pattern Recognition
Unsupervised Learning
Cluster the data based on their separation in the feature space.
K-means and fuzzy clustering methods
Supervised Learning
It uses labeled clusters of training samples in the feature space as models of classes.
Nearest neighbor classifier, which assigns a data point to the nearest class model in the feature space.

61. Nearest Neighbor Classifier

62. Statistical classification Method
A probabilistic approach can be applied to the task of classification to incorporate a priori knowledge to improve performance.
Bayesian and maximum likelihood methods have been widely used in object recognition and classification.
Bayesian

63. Statistical classification Method
The probability of a feature vector f belonging to the class i (ci)is denoted by p(ci /f).
The average risk of wrong classification for assigning the feature vector to the class cj is defined as
A Bayes classifier assigns an unknown feature vector to the class cj if

64. Feature and Image Classification
Rule-Based Systems
Analyzes the feature vector using multiple sets or rules that are designed to check specific conditions in the database of feature vectors to initiate an action.
The rules are composed of two parts
Condition premises
Actions
They are based on expert knowledge to infer the action if the conditions are satisfied.

65. Feature and Image Classification
A rule-based system has three sets of rules
Supervisory or strategy rules
Guide the analysis process and provide the control actions such as starting and stopping analysis.
Focus of attention rules
Bring specific features into analysis by accessing and extracting the required information or features from the database
Knowledge rules
Analyze the information with respect to the required conditions and implement an action causing changes in the output database.

66. A schematic diagram of a rule-based system for image analysis
Figure A schematic diagram of a rule-based system for image analysis.

67. Strategy Rules

68. FOA Rules

69. Knowledge Rules

70. Image and Feature Classification: Neural Networks
Several neural networks have been used for feature classification for object recognition and image interpretation.
Backpropagation
Radial Basis Function
Associative Memories
Self-Organizing Map

71. Neuro-Fuzzy pattern Classification
The process of network training could be seen as the attempt at finding an optimal dichotomy of the input space into these convex regions.
The classes are separated in the feature space by computing the homogeneous non-overlapping closed convex subsets.
The classification is obtained by placing separating hyperplanes between neighboring subsets representing classes.
Grohman and Dhawan
Fuzzy membership function Mf is dervised for each convex subset.
The classification decision is made by the output layer based on the “winner-take-all” principle.
The resulting category C is the convex set category with the highest value of membership function for the input pattern.

72. Convex sets Convex sets
A convex set is a set of elements from a vector space such that all the points on the straight line between any two points of the set are also contained in the set.
A set S in n-dimensional space is called a convex set if the line segment joining any pair of points of S lies entirely in S. If the set does not contain all the line segments, it is called concave.

73. Convex sets Convex Hull
The convex hull of a set of points is the smallest convex set that includes the points. For a two dimensional finite set the convex hull is a convex polygon.

74. Some basic Morphological Algorithm
Convex Hull
A set A is said to be convex if the straight line segment jointing any two points in A lies entirely within A.
The convex hull H of an arbitrary set S is the smallest convex set containing S.
The set difference H-S is called the convex deficiency of S.
The convex hull and convex deficiency are useful for object description.
Let Bi, i=1, 2, 3, 4, represent the four structuring elements in Fig (a).
The procedure consists of implementing the equation: where
Let Then the convex hull of A is
(9.5-4)
(9.5-5)

75. Some basic Morphological Algorithm
X indicates don’t care
The procedure consists of iteratively applying the hit-or-miss transform to A with B1; when no further changes occur, we perform the union with A and call the result D1.
The procedure is repeated with B2 until no further changes occur, and so on.
The union of the four resulting D’s constitutes the convex hull of A.

76. Some basic Morphological Algorithm

77. Convex sets Convex hull 演算法一: Jarvis's March (gift wrapping)
找出最下方的點 p0. 它一定在 convex hull 的邊界上.
找出 p1, 使 p0 與 p1 的連線與正 x 軸的夾角 (有向角) 最小.
找出 p2, 使 p2 與 p1 的連線與正 x 軸的夾角最小.
... 直到回到 p0 為止.

78. Convex sets Convex hull 演算法二: Graham's scan
找出最下方的點 p0. 它一定在 convex hull 的邊界上.
以「p0 到各點的射線與 x 軸的夾角」作為比較的依據, 對所有的點排序.
依序如下檢查 p1, p2,....
檢查 pi 時要做的事情: 看看 stack 上第二高的元素, stack 上最頂端的元素, 與 pi 三點兩射線是左轉還是右轉. 如果是右轉, 就 pop, 並重複此步驟.
push pi.
檢查下一個 pi.

79. Neuro-Fuzzy pattern Classification
The neuro-fuzzy pattern classifier design method includes three stages
Convex set creation
Hyperplane placement
hyperplane layer creation
Construction of the fuzzy membership function for each convex set.
Generation of the fuzzy membership function layer.

80. Neuro-Fuzzy pattern Classification
There are two requirements for computing the convex sets
Homogeneous
Need to devise a method of finding one-category points within another category’s hull.
How to find whether the point P lies inside of a convex hull (CH) of points.
How to find out if two convex hulls of points are overlapping.
Non-overlapping.

81. Neuro-Fuzzy pattern Classification
Algorithm A1 addresses the first problem using the separation theorem, which states that for two closed non-overlapping convex sets S1 and S2 there always exists a hyperplane that separates the two sets.
1. Consider P as origin.
2. Normalize points of CH
3. Find min and max vector coordinates in each dimension.
4. Find set E of all vectors V that have at least one extreme coordinate.
5. Compute mean and use it as projection vector f:

82. Neuro-Fuzzy pattern Classification
6. Set a maximum number of allowed iterations (usually=2n)
7. Find a set U=(u1, u2,…, um) (where m<=n) of all points in CH that have negative projection on f.
8. If U is empty (P is outside of CH) exit, else proceed to Step 9.
9. Compute coefficient y as:
10. Calculate correction vector by computing all of its k-dimensional components:
11. Update f: f=f-h.df, where h>1 is a training parameter.
12. If iteration limit exceed exit, otherwise go to step 7.

83. Neuro-Fuzzy pattern Classification
Algorithm A2: Convex subset creation
1. select one class category from the training set and consider all data points in the category.
2. Construct the convex subsets.
Add the current point P to the subset S.
Loop over points from negative category.
UpdateΛ.
3. If all points in the category have been assigned to a subset proceed to step 4, otherwise go back to Step 2 and create the next convex subset.
4. Check if all categories have been divided into convex subsets. If not, go back to Step 1 and create subsets of the next category.

84. Figure 8.18. The structure of the fuzzy membership function.

85. Figure 8.19. Convex set-based separation of two categories.

86. Figure 8.20. (a). Fuzzy membership function M1(x) for the subset #1
of the black category. (b). Fuzzy membership function M2(x) for the subset #2
of the black category.

87. Figure 8.22. Resulting decision surface Mblack(x) for the black category membership function
Figure Fuzzy membership function M3(x) (decision surface) for the white
category membership.

88. Image analysis example
It is difficult to distinguish between benign and malignant microcalcifications associated with breast cancer.
Dhawn used the second-order histogram statistics and wavelet processing to represent texture for classification into benign and malignant.
Two sets of ten wavelet features were computed for discrete Daubechies filter prototypes.
40 features were extracted and used in a Genetic algorithm based feature reduction and correlation analysis.
10 binary segmented microcalcification cluster features
10 global texture based image structure features.
20 wavelet analysis based local texture features.
(see Page. 242~245)

89. Image analysis example
Genetic algorithm were used to select the best subset of features from the binary cluster, global and local texture representation.
GA is a robust optimization and search method based on natural selection principles.
GA generate a population of individuals through selection, and search for the fittest individuals through crossover and mutation.
They operate on a representation of problem parameters, rather than manipulating the parameters themselves.
These parameters are typically encoded as binary strings that are associated with a measure of goodness, or fitness value.

90. Image analysis example
Through the process of reproduction, individual strings are copied according to their degree of fitness.
Once the parent population is selected through reproduction, the offspring population is created after application of genetic operators.
The purpose of crossover is to discover new regions of the search space rather than relying on the same population of strings.
In crossover, strings are probabilistically mated by swapping all characters located after a randomly chosen bit position.
Mutation is a secondary genetic operator that randomly changes the value of a string position to introduce variation in the population and recover lost genetic information.
Mutation preserves the random nature of the search process and regenerates fit strings that may have been destroyed or lost during crossover or reproduction.
The mutation rate controls the probability that a bit value will be changed.

91. Image analysis example
Using the GA algorithm, the initial set of 40 features was reduced to the two best correlated set of 20 features.
The selected features were used as inputs to the radial basis function for subsequent classification of the microcalcification.