Shape similarity and Visual Parts Longin Jan Latecki Temple Univ., In cooperation with Rolf Lakamper (Temple Univ.), Dietrich Wolter (Univ. of Bremen)

2 Object Recognition Process: Source: 2D image of a 3D object Matching: Correspondence of Visual Parts Contour Segmentation Contour Extraction Object Segmentation Evolution

3 What is this talk about ? Comparison of object shape based on object contours Object contours are naturally obtained in Computer Vision, Robot Navigation, and other applications as polylines (polygonal curves).  Shape similarity reduces to similarity of polylines. Shape similarity of polylines is not so simple: simple 1-1 vertex correspondence does not work a scale problem

4 only part of a polyline may be visible

5 Since polylines are obtained as boundary parts of objects in usually noisy sensor data (e.g., digital images): 1.two similar polylines do not need to have the same number of vertices, i.e., do not have to be of comparable level of detail, 2.do not have to be of comparable size, 3.may have only a subpart that is similar and that has a significant contribution to their shape = visual part Cognitive Similarity Requirements

6 Comparable level of detail (1) It can be achieved in the context of a single object or a group of objects (e.g., a query and a goal shape) How can we achieve a comparable level of detail for all objects if we treat each object separately? By placing each object on the same level of the Scale Space hierarchy.

7 Comparable level of detail  Scale Space

8 Discrete Curve Evolution (DCE) We achieve a comparable level of detail with DCE. Before a similarity measure is applied, the shape of objects is simplified by DCE in order to reduce influence of noise, simplify the shape by removing irrelevant shape features without changing relevant shape features.

Discrete Curve Evolution (DCE) u v w u v w It yields a sequence: P=P 0,..., P m P i+1 is obtained from P i by deleting the vertices of P i that have minimal relevance measure K(v, P i ) = |d(u,v)+d(v,w)-d(u,w)| >

10 Discrete Curve EvolutionDiscrete Curve Evolution: Preservation of position, no blurring

11 Discrete Curve EvolutionDiscrete Curve Evolution: robustness with respect to noise

12 Discrete Curve Evolution: extraction of linear segments

13 Discrete Curve Evolution: mathematical properties Convexity Theorem (trivial) Discrete curve evolution (when applied to a polygon) converges to a convex polygon. Continuity Theorem (nontrivial) Discrete curve evolution is continuous. L. J. Latecki, R.-R. Ghadially, R. Lakämper, and U. Eckhardt: Continuity of the discrete curve evolution. Journal of Electronic Imaging 9, pp , Polygon Recovery (nontrivial) DCE allows to recover polygons from their digital images. L.J. Latecki and A. Rosenfeld: Recovering a Polygon form Noisy Data. Computer Vision and Image Understanding (CVIU) 86, 1-20, 2002.

14 Comparable level of detail for DCE (=stop condition) is based on a threshold on the relevance measure

15 Comparable level of detail for DCE is based on a threshold on the relevance measure

16 Scale Space Approaches to Curve Evolution 1.reaction-diffusion PDEs 2.polygonal analogs of the PDE-evolution (Bruckstein et al. 1995) 3.approximation (e.g., Bengtsson and Eklundh 1991) Main differences: [to 1, 2:] Each vertex of the polygon is moved at a single evolution step, whereas in our approach the remaining vertices do not change their positions. [to 1, 3:] Our approach is parameter-free (we only need a stop condition)

17

18 (3)  Local similarity We need local similarity measure, i.e., we need to compare polylines but not polygons. Global similarity measures fail at: - partial visibility (occlusion) - not uniformly distributed noise - actually everything that occurs under real conditions

19 Local similarity A simple solution to local similarity measure: Given a target polylineT, find the most similar part P of polyline Q. Consider all subpolylines of Q  combinatorial explosion We consider only all connected subsets P of Q, O(n²) in the number of vertices n of Q. However, a connected subset P may be a distorted version of T.

20 Optimal Shape Similarity The optimal similarity of P to T is the similarity of modified P to T, where modified P is P with all features that make P distinct from T removed. Consider all subpolylines of P  combinatorial explosion s is a global similarity measure

21

22 All shape similarity measures presented in the literature are global measures. Although they can be applied to local parts, they are not optimal.

23 Presented approach A two-step solution: 1. We consider only all connected subsets P of Q, O(n²), n vertices of Q. 1. We use a greedy algorithm to compute optimal similarity, O(n²), n vertices of P. We need a global similarity measure s.

24 Shape Similarity in Tangent Space A polyline P is mapped to its turn angle function T(P) in the tangent space the height of each step shows the turn-angle, monotonically increasing intervals represent convex arcs, height-shifting corresponds to rotation, the resulting curve can be interpreted as 1D signal

25 Shape Similarity Measure: Arkin at al. PAMI 1991.

26 Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise

27 Shape Comparison: Contour Segmentation Solution: use this measure only locally, i.e., apply only to corresponding parts:

28 Shape Similarity

29

30 Correspondence of visual parts: Results

31 Correspondence of visual parts: non-rigid deformation L. J. Latecki and R. Lakämper: Shape Similarity Measure Based on Correspondence of Visual Parts. IEEE Trans. Pattern Analysis and Machine Intelligence 22, L. J. Latecki and R. Lakämper: Application of Planar Shape Comparison to Object Retrieval in Image Databases. Pattern Recognition 35, 2002.

32 Comparison to deformation energy

33 Partial Shape Matching This shape similarity measure works fine if the whole contour is given: Great performance in the MPEG-7 competition Life web-based shape search engineshape search engine Can it be applied when only contour parts are given? Yes, but only in the context of optimal shape similarity.

34 Subpart Selection We can use sliding window on the contour of the object Q to find a given target part T. However, we cannot expect to cut part P of Q that exactly corresponds to T, due to imperfect size of the window noise and change of view point scale selection. Therefore, a similarity measure must be able to overlook parts of Q whose shape is irrelevant w.r.t. Q. T Q

35

36 Matching and Simplification The main idea of the proposed computation of the optimal similarity is shape matching and simplification similar to DCE in one process. The main idea: We recursively remove a vertices of P to obtain P’ such that P’ is the most similar to T. os(T,P)= global minimum of s(T,P’). We achieve a comparable level of detail (1) for polyline P in the context of target T. Observe that this approach also solves the stop condition problem (DCE is applied to each object separately).

37 P T * =E

38

39 Matching and Simplification II Given a target T and a polyline P. os(T,P) = global minimum  S(T,P’) +  S(P’,P), where  and  are weights. The simplest weight assignment is  = 1 and  =0.

40

41

42 Applications Shape-based object recognition Retrieval in image and video databases Object tracking Shape-based tracking of objects in laser scans Robot mapping Robot localization in top view maps

43 Shape-based tracking of objects in laser scans An example top view image of LRF scan data

44 Shape-based tracking of objects in laser scans Row scan data (left) is segmented into polyline (right)

45 Scan polylines simplification with DCE

46 Matching scans by shape Since shape may be very simple, we also use cyclic order and proximity information

47 Shape-based tracking of objects in laser scans Building a global map Demo movie Why go beyond the simple proximity of points computed by minimizing the least squared distance of all points? Robot may slip or turn (unconsciously): Shape and order of polylines remain similar even if the displacement is significant

48 Robot mapping and localization Shape similarity useful for: globally consistent mapping localization when odometry is not given or unreliable

49 Future Work Learning of visual parts: How to select the most distinctive parts of objects? Our approach is based on statistics and our shape similarity measures: We use statistical methods to find the smallest possible set of most different parts within a given class of objects and the smallest possible set of most separating parts among different object classes. Previous approaches to find visual parts are based on differential geometry. They are static in that the parts will be always the same for different classes of objects. Our approach is dynamic: selected parts depend on the objects seen.

50 Automatically Selected Parts