Computer and Robot Vision I Chapter 2 Binary Machine Vision: Thresholding and Segmentation Presented by: 傅楸善 & 翁丞世 0925070510 r04945028@ntu.edu.tw 指導教授: 傅楸善 博士 Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C.

2.1 Introduction To detect the objects, the input image is often simplified. binary value 1: considered part of object binary value 0: background pixel binary machine vision: generation and analysis of binary image DC & CV Lab. CSIE NTU

2.2 Thresholding 𝐵 𝑟,𝑐 =1, 𝑖𝑓 𝐼 𝑟,𝑐 ≥𝑇 𝐵 𝑟,𝑐 =0, 𝑖𝑓 𝐼 𝑟,𝑐 <𝑇 r : row, c: column I: grayscale intensity, B: binary intensity T: intensity threshold DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding The question of thresholding is a question of the automatic determination of the threshold value. DC & CV Lab. CSIE NTU

2.2 Thresholding histogram ℎ 𝑚 =#{(𝑟,𝑐)|𝐼 𝑟,𝑐 =𝑚} m spans each gray level value e.g. 0 - 255 #: operator counts the number of elements in a set DC & CV Lab. CSIE NTU

2.2 Thresholding 2 3 1 2-bits source image histogram of left image 2 3 1 2-bits source image histogram of left image DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding Bayonet Neill Concelman Connector (BNC) The standard connector used to connect IEEE 802.3 10Base2 coaxial cable to an MAU(媒體附著單元 (Media attachment unit)) DC & CV Lab. CSIE NTU

2.2 Thresholding DC & CV Lab. CSIE NTU

2.2 Thresholding Minimizes a criterion function Minimize the within-group variance (Otsu’s method) Minimize the Kullback information distance DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance 𝑃 1 , …, 𝑃 𝐼 : histogram probabilities of gray values 1 … I 𝑃 𝐼 = # 𝑟,𝑐 𝐼𝑛𝑡𝑒𝑛𝑠𝑖𝑡𝑦 𝑟,𝑐 =𝐼 𝑅×𝐶 𝑅×𝐶: the spatial domain of the image 𝑅: the number of row of the image 𝐶: the number of column of the image DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Observations A group is a set of pixels with intensity homogeneity. Homogeneity is measured by the use of variance High homogeneity group has low variance Low homogeneity group has high variance Objective Select a dividing score such that the weighted sum of the within-group variances is minimized. DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Given threshold t, divide histogram into two parts. Part 1 is the left hand side of the histogram. Part 2 is the right hand side of the histogram. DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Within-group variance 𝜎 𝑤 2 : weighted sum of group variances 𝜎 𝑊 2 = 𝑞 1 𝑡 𝜎 1 2 𝑡 + 𝑞 2 𝑡 𝜎 2 2 𝑡 Find 𝑡 ∗ which minimizes 𝜎 𝑊 2 𝑡 𝑡 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 𝜎 𝑊 2 (𝑡) DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Implementation For i from 0 to 255: Compute 𝑞 1 𝑡 , 𝑞 2 𝑡 , 𝜎 1 2 𝑡 , 𝜎 2 2 𝑡 Compute 𝜎 𝑤 2 𝑡 = 𝑞 1 𝑡 ∗ 𝜎 1 2 𝑡 + 𝑞 2 𝑡 ∗ 𝜎 2 2 𝑡 If 𝜎 𝑤 2 𝑡 ∗ > 𝜎 𝑤 2 𝑡 : 𝑡 ∗ =𝑡 DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Speed up 𝜎 2 = 𝑖=0 𝑡 𝑖−𝜇 2 𝑃 𝑖 + 𝑖=𝑡+1 𝐼 𝑖−𝜇 2 𝑃 𝑖 = 𝑖=0 𝑡 𝑖− 𝜇 1 𝑡 + 𝜇 1 𝑡 −𝜇 2 𝑃 𝑖 + 𝑖=𝑡+1 𝐼 𝑖− 𝜇 2 𝑡 + 𝜇 2 𝑡 −𝜇 2 𝑃 𝑖 = 𝑖=0 𝑡 [ 𝑖− 𝜇 1 𝑡 2 + 𝑖− 𝜇 1 𝑡 ∗ 𝜇 1 𝑡 −𝜇 + 𝜇 1 𝑡 −𝜇 2 ]𝑃 𝑖 + 𝑖=𝑡+1 𝐼 [ 𝑖− 𝜇 2 𝑡 2 + 𝑖− 𝜇 2 𝑡 ∗ 𝜇 2 𝑡 −𝜇 + 𝜇 2 𝑡 −𝜇 2 ]𝑃 𝑖 Σ 𝑖− 𝜇 1 𝑡 ∗𝑃 𝑖 =0 Σ 𝑖− 𝜇 2 𝑡 ∗𝑃 𝑖 =0 DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Speed up 𝜎 2 = 𝑖=0 𝑡 [ 𝑖− 𝜇 1 𝑡 2 + 𝜇 1 𝑡 −𝜇 2 ]𝑃 𝑖 + 𝑖=𝑡+1 𝐼 [ 𝑖− 𝜇 2 𝑡 2 + 𝜇 2 𝑡 −𝜇 2 ]𝑃 𝑖 𝑖=𝑡+1 𝐼 [ 𝑖− 𝜇 2 𝑡 2 + 𝜇 2 𝑡 −𝜇 2 ]𝑃 𝑖 = 𝑖=0 𝑡 𝑖− 𝜇 1 𝑡 2 𝑃 𝑖 + 𝜇 1 𝑡 −𝜇 2 𝑞 1 + 𝑖=𝑡+1 𝐼 𝑖− 𝜇 2 𝑡 2 𝑃 𝑖 + 𝜇 2 𝑡 −𝜇 2 𝑞 2 𝑖=𝑡+1 𝐼 𝑖− 𝜇 2 𝑡 2 𝑃 𝑖 + 𝜇 2 𝑡 −𝜇 2 𝑞 2 = 𝜎 1 2 ∗ 𝑞 1 + 𝜎 2 2 ∗ 𝑞 2 + 𝜇 1 𝑡 −𝜇 2 𝑞 1 + 𝜇 2 𝑡 −𝜇 2 𝑞 2 = 𝜎 𝑤 2 + 𝑞 1 𝑡 1− 𝑞 1 𝑡 [ 𝜇 1 𝑡 − 𝜇 2 𝑡 ] 𝜇= 𝑞 1 (𝑡)∗ 𝜇 1 (𝑡)+ 𝑞 2 (𝑡)∗ 𝜇 2 (𝑡) 𝑞 2 𝑡 =1− 𝑞 1 (𝑡) DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Speed up 𝜎 2 = 𝜎 𝑤 2 + 𝑞 1 𝑡 1− 𝑞 1 𝑡 [ 𝜇 1 𝑡 − 𝜇 2 𝑡 ] 𝑡 ∗ =𝑎𝑟𝑔𝑚𝑎𝑥 𝑞 1 (𝑡)[1− 𝑞 1 𝑡 ][ 𝜇 1 𝑡 − 𝜇 2 𝑡 ] Fortunately, we can calculate 𝑞 1 𝑡 , 𝜇 1 𝑡 , 𝜇 2 𝑡 recursively. constant minimize maximize DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance Speed up Fortunately, we can calculate 𝑞 1 𝑡 , 𝜇 1 𝑡 , 𝜇 2 𝑡 recursively. 𝑞 1 𝑡+1 = 𝑞 1 𝑡 +𝑃 𝑡+1 𝜇 1 𝑡+1 = 𝜇 1 (𝑡)∗ 𝑞 1 𝑡 + 𝑡+1 ∗𝑃 𝑡+1 𝑞 1 (𝑡+1) 𝜇 2 𝑡+1 = 𝜇− 𝜇 1 𝑡+1 ∗ 𝑞 1 𝑡+1 1− 𝑞 1 (𝑡+1) DC & CV Lab. CSIE NTU

2.2.1 Minimizing Within-Group Variance DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance  Gaussian distribution DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance Assumption The observations come from a weighted mixture of two Gaussians distributions. Gaussian Distribution of Background Gaussian Distribution of Object DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance Objective Formulation Determine a threshold T that results in two Gaussian distributions which minimize Kullback divergence P(I) : observed histogram distribution f(I) : a mixture of Gaussian distributions determined by T DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance constant minimize     DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance Implementation For i from 0 to 255: Compute 𝑞 1 𝑡 , 𝑞 2 𝑡 , 𝜎 1 2 𝑡 , 𝜎 2 2 𝑡 Compute 𝐻(𝑡) If 𝐻( 𝑡 ∗ )>𝐻(𝑡): 𝑡 ∗ =𝑡 DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance DC & CV Lab. CSIE NTU

2.2.2 Minimizing Kullback Information Distance Left dark line: Within-Group Variance (Otsu) Right dark line: Kullback Information (Kittler-Illingworth) DC & CV Lab. CSIE NTU

2.3 Connected Components Labeling Description Connected Components labeling is a grouping operation. It performs the unit change from pixel to region or segment. All pixels are given the same identifier Have value binary 1 Connect to each other Definition DC & CV Lab. CSIE NTU

2.3 Connected Components Labeling Terminology label: unique name or index of the region connected components labeling: a grouping operation pixel property: position, gray level or brightness level region property: shape, bounding box, position, intensity statistics Definition DC & CV Lab. CSIE NTU

2.3 Connected Components Labeling Definition of Connected Component Two pixels 𝑝 and 𝑞 belong to the same connected component 𝐶 if there is a sequence of 1-pixels ( 𝑝 0 , 𝑝 1 ,…, 𝑝 𝑛 ), where 𝑝 0 =𝑝 𝑝 𝑛 =𝑞 𝑝 𝑖−1 , 𝑝 𝑖 :𝑖=1,…,𝑛 are neighbor Definition DC & CV Lab. CSIE NTU

2.3.1 Connected Components Operators To explain what neighborhood is DC & CV Lab. CSIE NTU

2.3.1 Connected Components Operators 1 1 2 3 1 2 source binary image example Connected components labeling with 4-adjacency Connected components labeling with 8-adjacency DC & CV Lab. CSIE NTU

2.3.2 Connected Components Algorithms All the algorithms process a row of the image at a time All the algorithms assign new labels to the first pixel of each component and attempt to propagate the label of a pixel to right or below neighbors Principle & stop condition DC & CV Lab. CSIE NTU

2.3.2 Connected Components Algorithms What label should be assigned to pixel A? How to keep track of the equivalence of two or more labels? How to use the equivalence information to complete processing? Issue A應該放什麼標籤 怎麼保持追蹤等價的標籤 怎麼使用等價訊息去完成處理 DC & CV Lab. CSIE NTU

2.3.2 Connected Components Algorithms An Iterative Algorithm The Classical Algorithm *A Space-Efficient Two-Pass Algorithm with a Local Equivalence Table *An Efficient Run-Length Implementation of the Local Table Method DC & CV Lab. CSIE NTU

2.3.3 An Iterative Algorithm Implementation Initialize each pixel with an unique label Changed = True While Changed == True iteration of top-down followed by bottom-up passes Changed = check is any label changed DC & CV Lab. CSIE NTU

2.3.3 An Iterative Algorithm Why can’t just one iterative Jump to 35(ori) 36(after top-down)=>12與7的交界處 DC & CV Lab. CSIE NTU

2.3.4 The Classical Algorithm performs label propagation as above when two different labels propagate to the same pixel, the smaller label propagates and equivalence entered into table resolve equivalence classes by DFS (Depth-First Search) second pass performs a translation Two top-down phase First phase: assign label and establish equ_table mid-phase: find equivalence classes Second phase: chx label according to equ_label DC & CV Lab. CSIE NTU

2.3.4 The Classical Algorithm Example Two top-down phase First phase: assign label and establish equ_table mid-phase: find equivalence classes Second phase: chx label according to equ_label {2=4} {3=5} {1=5} DC & CV Lab. CSIE NTU

2.3.4 The Classical Algorithm Example Resolve equivalence {2=4} {3=5} {1=5} {2=4} {1=3=5} Two top-down phase First phase: assign label and establish equ_table mid-phase: find equivalence classes Second phase: chx label according to equ_label DC & CV Lab. CSIE NTU

2.3.4 The Classical Algorithm Pros More efficient, only two passes Cons Need more memory to store equivalence table Two top-down phase First phase: assign label and establish equ_table mid-phase: find equivalence classes Second phase: chx label according to equ_label DC & CV Lab. CSIE NTU

2.3.5 A Space-Efficient Two-Pass Algorithm That Uses a Local Equivalence Table Small table stores only equivalences from current preceding line Relabel each line with equivalence labels when equivalence detected Point2. alternation DC & CV Lab. CSIE NTU

2.3.5 A Space-Efficient Two-Pass Algorithm That Uses a Local Equivalence Table DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis Signature analysis perform a unit change from the pixel to the region or segment. It was firstly used in character recognition. The signature, which is a projection, is the histogram of the non-zero pixels of the masked image. projections can be vertical, horizontal, diagonal, circular… DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis General Signatures Vertical Projection Horizontal Projection Diagonal Projection ℎ 𝑟 =⋕# 𝑥,𝑦 𝑥,𝑦 =1,𝑥=𝑟 𝑣 𝑟 =⋕{ 𝑥,𝑦 | 𝑥,𝑦 =1,𝑦=𝑐} DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis Steps Thresholding: generate the binary image. Projection Computation: compute the vertical, horizontal, or diagonal projections. Projection Segmentation: divide the image into several segments or regions according to the signatures. DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis segment the vertical and horizontal projections treat each rectangular subimage as the image DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis Magnetic ink character recognition 光學字母辨識 磁性墨水辨識 OCR: Optical Character Recognition MICR: Magnetic Ink Character Recognition DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis Diagonal projections: : from upper left to lower right : from upper right to lower left object area: sum of all the projections values in the segment DC & CV Lab. CSIE NTU

2.4 Signature Segmentation and Analysis DC & CV Lab. CSIE NTU

DC & CV Lab. CSIE NTU

2.5 Summary Create binary image by thresholding. Find out connected component on binary image. Generate signature segmentation by projection. DC & CV Lab. CSIE NTU

Project due Oct. 4 Write a program to generate: a binary image (threshold at 128) a histogram connected components (regions with + at centroid, bounding box) DC & CV Lab. CSIE NTU