OP2: Information Theoretical Region Merging Approaches and Fusion of Hierarchical Image Segmentation Results Felipe Calderero, Thesis Advisor: Ferran Marqués Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 21 st October 2010

Outline 1. Introduction 2. Information Theoretical Region Merging 3. Cooperative Region Merging 4. Conclusions Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 2

1. What is image segmentation?  Image Segmentation  Partition of the image into regions (disjoint sets of spatially contiguous pixels)  Key step in image analysis  Semantically, first level of abstraction  Practically, reduction of primitives  But image segmentation is a difficult task… Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 3

1. An ill-posed problem…  Image Segmentation is an ill-posed problem  A unique solution may not exist  Different levels of detail  Same level of detail Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 4

1. An ill-posed problem…  Image Segmentation is an ill-posed problem  A unique solution may not exist  Different levels of detail  Same level of detail Hierarchical Segmentation Approaches Fusion of (Hierarchical) Segmentation Results Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 5

1. PhD Thesis Objectives  Objective 1: Hierarchical Segmentation Approaches  Objective 2: Fusion of (Hierarchical) Segmentation Results Provide an unsupervised hierarchical solution to the segmentation of generic images Design a generic and scalable segmentation scheme to fuse in an unsupervised manner hierarchical segmentation results Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 6

1. PhD Thesis Approach  Solutions to Objective 1 and Objective 2  Generic  No a priori information  Hierarchical solution  Unsupervised Bottom-up hierarchy [Marr82] Region Merging Techniques Hierarchy of most representative partitions at different levels of detail Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 7

1. PhD Thesis Contributions  Contribution 1: Hierarchical Segmentation Approaches  Contribution 2: Fusion of (Hierarchical) Segmentation Results Information Theoretical Region Merging Techniques (IT-RM) Cooperative Region Merging Scheme (CRM) Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 8

2. Region Merging Techniques  Region Merging  Hierarchical bottom-up segmentation approaches  Specified by  Region Model  Merging Criterion  Merging Order  Partition Selection Criterion G A A B B C C D D E E F F G G Binary Partition Tree (BPT) [Garrido99] Efficiency of computation and representation Selection Criterion Hierarchy creation Relevant partition extraction Unsupervised mode Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 9

2. Information Theory Region Merging  Information Theoretical Region Merging (IT-RM)  Statistical and information theoretical framework 1.Region Model  i.i.d / Markov region model 2.Merging Criteria  Kullback-Leibler / Bhattacharyya Criteria 3.Merging Order  Classical / Scale-based 4.Unsupervised mode:  Multiple partition selection criterion (statistically relevant) Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 10

2. IT-RM Applications  Semantic image analysis Original 1 st Significant Partition 2 nd Significant Partition Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 11

2. IT-RM Applications  Semantic image analysis (textures) Original 1 st Significant Partition 2 nd Significant Partition Barcelona Forum on Ph.D. Researh in Communications, Electronics and Signal Processing 12

2. IT-RM Applications  Object-based representation and analysis Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 13

3. Cooperative Region Merging  Motivation: Objective 2  Most IT-RM techniques have similar and accurate performance…  Instead of selecting, why not combining the set of techniques?  Cooperative Region Merging (CRM)  Similar to a negotiation process in decision making Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 14

Characteristics: parallel, scalable, hierarchical, unsupervised, flexible… 3. Cooperative Region Merging  Cooperative Region Merging 1. Segmentation results are computed independently by each technique (RM step) 2. A basic consensus or agreement is established between the set of techniques (FUSION step) 3. Steps 1 and 2 are repeated while further consensus is possible Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 15

3. CRM Applications  Accuracy and robustness improvement  Combining different segmentation techniques 1 st Median Partition 2 nd Median Partition 3 rd Median Partition Original Human Partition Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 16

3. CRM Applications  Fusion of heterogeneous information channels  Combining color and depth for object-based segmentation Color image Disparity map 1 st Sign. Partition 2 nd Sign. Partition 3 rd Sign. Partition Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 17

3. CRM Applications  Scalability and flexibility of the fusion scheme  Fusion of multispectral band and vegetation classification Vegetation extraction using bands: B, G, R, IRNDVIRGB composition Vegetation extraction using bands: B, G, R, IR + PAN Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 18

4. Conclusions  Unsupervised segmentation of generic images: a challenge  Hierarchical Region Merging approach as a possible solution  Info. Theoretical Region Merging Segmentation  State-of-the-art results in unsupervised manner  Relevant image explanations at different levels of detail  Application independent object-based semantic tool  Cooperative Region Merging Information Fusion  Accuracy and robustness improvement in unsupervised manner  Scalable, flexible and generic scheme  Fusion of homogeneous/heterogeneous information channels Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 19

Information Theoretical Region Merging Approaches and Fusion of Hierarchical Image Segmentation Results Felipe Calderero Image Processing Group Pompeu Fabra University (UPF) Barcelona, Spain Ferran Marqués Image Processing Group Universitat Politècnica de Catalunya Barcelona, Spain

Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 21

4. Applications  CRM: Fusion of color and depth information Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 22

4. Applications  IT-RM: Semantic image analysis Original 1 st Significant Partition 2 nd Significant Partition Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 23

5. Conclusions  Information Theoretical Region Merging  State-of-the-art segmentation results without any assumption about the nature of the region  Unsupervised extraction of most relevant image explanations at different levels of detail  Application independent accurate tool for object-based representation and semantic analysis of generic images Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 24

5. Conclusions  Cooperative region merging  Global improvement of the accuracy and the stability of the segmentation results by combining different segmentation approaches  Parameter removal solution  Fusion of heterogeneous information channels  Flexibility to incorporate specificities of the fusion problem  A priori information about the fused sources (e.g. channel priority)  Joint segmentation and classification stages Barcelona Forum on Ph.D. Research in Communications, Electronics and Signal Processing 25