Student: Chih-Wei Fang ( 方志偉 ) Adviser: Jenn-Jier James Lien ( 連震杰 ) Robotics Laboratory, Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, Taiwan, R.O.C. Texture Analysis and Synthesis Using Directional Patch-Based Sampling

Motivation 2

Related Work Onion-peel approach [5] A. Criminisi, P. Perez, and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting,” IEEE Trans. on Image Processing, Vol. 13, No. 9, pp ,

Related Work [5] A. Criminisi, P. Perez, and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting,” IEEE Trans. on Image Processing, Vol. 13, No. 9, pp , P(p) = C(p)D(p) Confidence term Data term 4

Problems of Ref [5] Larger removal region is not easy to converge. Influenced by high-frequency components or noise. Filling in the matching patch from searching exhaustively. 5

System Framework – Multi-resolution Down-sampling for Training The training process extending from the input image to get the weight vector of each patch. Up-sampling for Image Completion Directional image completion to propagate the structure. Non-directional image completion to maintain detailed features. 6

Down-sampling from levels 0 to L I 1 I 2 I 3 I L=4 α 1 α 2 α 3 α L=4 Input image I 0 Inverse matte α 0 Down-sampling for Training Total M patches in background regions from levels 1 to L … … Select O-shaped patterns for training … … Create eigenspace Ψ Project the M O-shaped patterns onto the first N eigenvectors to have corresponding weight vectors. Cluster weight vectors using VQ. 7

Sampling It is unnecessary to use entire patches of image I 0. It will include many unnecessary patches. Require a large amount of template matching operations. Image I 0 contains more noises affect the result of the PCA process. O-shaped pattern insteads of whole patch for the training data. Whole patch elements to the training data for further matching comparison may result in a discontinuous structure of the patch. Increase the training time certainly. Search PatchO-shaped Pattern O-shaped Pattern Vector Wp Hp ω K pixels … 8

PCA Reduce the dimensions of the data representation 9

Recombine the features of the O-shaped pattern First several eigenvectors => control the global geometrical structure. The middle eigenvectors => control the local detailed features. The last few eigenvectors => control some noises. Eigenvectors 10

Projection and VQ Projecting O-shaped Patterns onto Eigenspace Ψ Reducing dimensions from K to N Vector quantization (VQ) is adopted for clustering the projection weight vectors in the eigenspace so as to reduce the comparison time. 11

System Framework Down-sampling for Training The training process extending from the input image to get the weight vector of each patch. Up-sampling for Image Completion Directional image completion to propagate the structure. Non-directional image completion to maintain detailed features. 12

Level L-1: Directional and Non-Directional IC 1. Starting from patch having larger Hessian matrix decision value (HMDV) 2. If (HMDV) ≧ Threshold: Then Search along the 1st eigenvector v 1 direction of Hessian matrix (directional). Else Texture synthesis (non-directional). Up-sampling for Image Completion Level L: Initialize synthesis values. Levels L-2 to 1: Non-Directional IC 1. Starting from patch having larger HMDV. 2. Texture synthesis. Level 0: Texture Refinement 1. Starting from patch having larger HMDV. 2. Texture synthesis. 3. Find the best matching patch from image I 0. FiFi BiBi v1v1 λ 1 I0I0 I i, i=1~L FiFi BiBi Decision window 13

Filling Order Hessian Matrix => Obtain λ 1 and λ 2 (λ 1 ≧ λ 2 ) Hessian Matrix Decision Value 14

Hessian Matrix Decision Value Higher HMDV value V (V >> 1.0 or λ 1 >> λ 2 ) directional and exists stronger edge along eigenvector v 1. If the HMDV value V is close to or equal to 1 If both λ 1 and λ 2 have higher values More detailed features or high-frequency noises. If both λ 1 and λ 2 have lower values Smooth patch. 15

Hessian Matrix Decision Value 16

Hessian Matrix Decision Value 17

Level L-1: Directional and Non-Directional IC 1. Starting from patch having larger Hessian matrix decision value (HMDV) 2. If (HMDV) ≧ Threshold: Then Search along the 1st eigenvector v 1 direction of Hessian matrix (directional). Else Texture synthesis (non-directional). Up-sampling for Image Completion Level L: Initialize synthesis values. Levels L-2 to 1: Non-Directional IC 1. Starting from patch having larger HMDV. 2. Texture synthesis. Level 0: Texture Refinement 1. Starting from patch having larger HMDV. 2. Texture synthesis. 3. Find the best matching patch from image I 0. FiFi BiBi v1v1 λ 1 I0I0 I i, i=1~L FiFi BiBi Decision window 18

Level L-1: Directional and Non-Directional IC 1. Starting from patch having larger Hessian matrix decision value (HMDV) 2. If (HMDV) ≧ Threshold: Then Search along the 1st eigenvector v 1 direction of Hessian matrix (directional). Else Texture synthesis (non-directional). Up-sampling for Image Completion Level L: Initialize synthesis values. Levels L-2 to 1: Non-Directional IC 1. Starting from patch having larger HMDV. 2. Texture synthesis. Level 0: Texture Refinement 1. Starting from patch having larger HMDV. 2. Texture synthesis. 3. Find the best matching patch from image I 0. FiFi BiBi v1v1 λ 1 I0I0 I i, i=1~L FiFi BiBi Decision window 19

Texture Synthesis The O-shaped pattern of each search patch is projected onto the eigenspace Ψ to obtain the corresponding weight vector. Based on the similarity measure of Euclidean distance between the weight vector of this search pattern and those of the cluster centers, this search pattern will be classified to the nearest cluster. Then this search pattern will compare with all patterns within the same cluster to find the best matching pattern. The patch corresponding to the best matching pattern will be directly pasted onto the location of the search patch. 20

Level L-1: Directional and Non-Directional IC 1. Starting from patch having larger Hessian matrix decision value (HMDV) 2. If (HMDV) ≧ Threshold: Then Search along the 1st eigenvector v 1 direction of Hessian matrix (directional). Else Texture synthesis (non-directional). Up-sampling for Image Completion Level L: Initialize synthesis values. Levels L-2 to 1: Non-Directional IC 1. Starting from patch having larger HMDV. 2. Texture synthesis. Level 0: Texture Refinement 1. Starting from patch having larger HMDV. 2. Texture synthesis. 3. Find the best matching patch from image I 0. FiFi BiBi v1v1 λ 1 I0I0 I i, i=1~L FiFi BiBi Decision window 21

Experimental Results Image Inpainting Bertalmio et al. [2] ACM SIGGRAPH Exemplar-Based Image Inpainting Criminisi et al. [5] IEEE Trans. on Image Processing Our Method Input Image C4C4 C3C3 C2C2 C1C1 C0C0 22

Experimental Results Image Inpainting Bertalmio et al. [2] ACM SIGGRAPH Exemplar-Based Image Inpainting Criminisi et al. [5] IEEE Trans. on Image Processing Our MethodInput Image 23

Experimental Results Our MethodInput Image 24 Exemplar-Based Image Inpainting Criminisi et al. [5] IEEE Trans. on Image Processing Image Inpainting Bertalmio et al. [2] ACM SIGGRAPH

Experimental Results Processing Time Our Method Exemplar-Based Image Inpainting Criminisi et al. [5] Image Inpainting Bertalmio et al. [2] Windmill11 Second139 Second2 Second Diving2 Second50 Second1 Second Slope4 Second61 Second1 Second 25

Conclusions Multi-Resolution Accelerate the convergence of the system and Handle the large removed region More training patches with various scales Hessian Matrix Decision Directional image completion to propagate the structure Non-directional image completion to maintain detailed features. Training for input image Reduce the computational time during the similarity measure at the synthesis process. 26

Thanks! 27