1 Segmentation of Salient Regions in Outdoor Scenes Using Imagery and 3-D Data Gunhee Kim Daniel Huber Martial Hebert Carnegie Mellon University Robotics Institute

2 Perception for Unmanned Vehicles Imagery 3-D scan Sensors Input Data Vrml_file

3 Problem Statement Our ultimate goal is… Here, we focus on a crucial pre-requisite step !!

4 Object Detection Naïve Scanning

5 Object Detection Prioritize the searching regions Salient Regions

6 Problem Statement Saliency Detection using Imagery and 3-D Data – A Mid-level vision task – No high-level priors, models, or learning – Only low-level information – Ex) pixel colors, ( x, y, z )-coordinates

7 Problem Statement 1. An image 2. 3D scan data Information-theoretic optimal clustering Segmentation of top-k most salient regions

8 Saliency Saliency: The quality of standing out relative to neighboring items Top-down – Driven by high-level concepts – Memories and Experiences Bottom-up – Driven by low-level features – Intensity, contrast, color, orientation, and motion

9 Two Bottom-Up Models Itti-Koch-Niebur Model [1] [1] L.Itti, C.Koch and E.Niebur A Model of Saliency-based Visual Attention for Rapid Scene Analysis, PAMI, 1998.

10 Two Bottom-Up Models Kadir-Brady Saliency detector [3] – Find the region which are locally complex, and globally discriminative. [3] T.Kadir and M. Brady, Scale, Saliency and Image Description. IJCV 45 (2):83-105, More flatter, higher complex Scale, rotation, affine invariance

11 Previous Work for Saliency in 3-D [1] S. Frintrop, E. Rome, A. N¨uchter, and H. Surmann. A bimodal laser-based attention system. CVIU, [2] D. M. Cole, A. R. Harrison, and P.M. Newman. Using naturally salient regions for slam with 3d laser data, ICRA workshop on SLAM, BILAS [3]: Itti et al [1]’s model Cole et al. [4]: Kadir-Brady [2]’s model

12 Saliency of 3-D Data A point cloud Gestalt laws of grouping [5] Proximity laws Continuity & Simplicity laws

13 Saliency of 3-D Data How to detect salient regions in 3-D data? (Answer) Find the set of clusters that best fit 3-D pdfs – Gaussian – Uniform GaussianUniform

14 Robust Information-Theoretic Clustering (RIC) [6] Input: A feature set + Families of pdfs Output: Clusters according to how well they fit the pdfs Minimum Description Length (MDL) principle – Goodness of fit = compression costs – Huffman-like coding [6] C.Bohm,C.Faloutsos, J.Y.Pan, and C.Plant, Robust information-theoretic clustering, KDD 2006

15 Proposed Approach Clustering of 3-D Data using RIC Projection of Clusters on an Image Compute Saliency Values of Clusters Image Segmentation of Top-ranked Saliency Regions

16 Clustering in 3-D Data RIC clustering Uniform dist Gaussian dist Information Theoretic Optimal Clustering

17 Proposed Approach Clustering on 3-D Data using RIC Projection of Clusters to an Image Compute Saliency Values of Clusters Image Segmentation of Top-ranked Saliency Regions Projection of Clusters to an Image

18 Saliency Features 3D-dataImage data (RGB color) LocalRegionalGlobal

19 Saliency Features Local 3D-data 1. Compression Cost - how well the cluster is fit to the reference family of pdfs

20 Saliency Features Local RGB-data 2. Entropy of RGB histograms [7] - Follow Kadir-Brady’s definition [7] T. Kadir and M. Brady. Saliency, scale and image description. IJCV, 45(2):83–105, 2001.

21 Saliency Features Regional RGB-data 3. Center-surround contrast [8] - Follow Itti et al’s definition [8] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum. Learning to detect a salient object, CVPR, 2007.

22 Saliency Features Global RGB-data 4. Color spatial distribution [8] - Rare color: More salient [8] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum. Learning to detect a salient object, CVPR, 2007.

23 Proposed Approach Clustering on 3-D Data using RIC Projection of Clusters to an Image Compute Saliency Values of Clusters Image Segmentation of Top-ranked Saliency Regions Projection of Clusters to an Image

24 Image Segmentation From Sparse Points to Dense Regions Using Conventional Markov-Random Field (MRF) Models – Labeling Problem

25 Qualitative Results

26 Qualitative Results

27 Future Work Integration with the Recognition Over-segmentation & Under-segmentation – Model-specific pdfs

28 Conclusion Bottom-up saliency detection using Imagery and 3-D data as a mid-level task – (x,y,z)-coordinates of 3-D data and Colors at pixels. Practically useful building block for perception of unmanned vehicles