Shape Classification Using the Inner-Distance Haibin Ling David W. Jacobs IEEE TRANSACTION ON PATTERN ANAYSIS AND MACHINE INTELLIGENCE FEBRUARY 2007

Outline Introduction Related work Inner-Distance Articulation invariant signatures Inner-distance shape context Shortest path texture context Experiments

Introduction We use the inner-distance to build shape descriptors that are robust to articulation and capture part structure. Inner-distance is defined as the length of the shortest path between landmark points within the shape boundary.

Related work Three categories to handle parts classification:  Statistical methods to describe the articulation between parts and often require a learning process to find the model parameters.  To measures the similarity between shapes between shapes via part-to-part matching and junction parameter distribution.  To capture the part structure by considering the interior of shape boundary. Our method belongs to. Skeleton-based approaches

The inner-distance: The definition Define a shape as a connected and closed subset of R 2. Given a shape and two points,the inner-distance between denoted as,is defined as the length of the shortest path connecting and within. Note: (1) In rare case where there are multiple shortest paths, we arbitrarily choose one. (2) Shapes are defined by their boundary, hence only boundary points are used as landmark points.

The inner-distance: Computation Shortest path algorithms:  (1) Build a graph with the sample points. For each pair of sample points p 1 and p 2, if the line segment connecting p 1 and p 2 fall entirely within the object, Let an edge between p 1 and p 2 is added to the graph with its weight equal to the Euclidean distance || p 1 - p 2 ||. Note: (1)Neighboring boundary points are always connected. (2)(2) The inner-distance reflects the existence of holes without using samples points from hole boundary.

The inner-distance: Computation  (2) find the inner-distance between all pairs of points according to the graph. The whole computation takes.  It takes time to checi whether a line segment between two points is inside the given shape.  The complixity of graph construction is of.

The inner-distance: A model of articulate objects Articulated objects. (a) An articulated shape. (b) Overlapping junstions. (c) Ideal articulation.

The inner-distance: A model of articulate objects is constant and very small compared to the size of the articulated parts. An articulated to another articulated object is one-to-one continuous mapping.

The inner-distance: articulation insensitivity Changes of the inner-distance are due to junction deformations. That means change is very small compared to the size of parts.

The inner-distance: articulation insensitivity Theorem: Proof: Is decomposed into segments.

The inner-distance: articulation insensitivity

Example: Note: is not the shortest path.

The inner-distance: ability to capture structures It is hard to prove because no clear part decomposition. Show how the inner-distance capture part structure with examples:

The inner-distance: ability to capture structures With about the same number of sample points, the four shapes are virtually indistinguishable using distribution of Euclidean distance. However, their distributions of the inner-distance are quite different except for the first two shapes. Note: more sample points will not affect the above statement.

Articulation Invariant Signatures The inner-distance is used to build articulation invariant signatures for 2D shapes using multidimensional scaling (MDS). Given sample points on the shape O. the inner-distance. MDS finds the transformed points such that the Euclidean distance minimize the stress S(Q)

Articulation Invariant Signatures Example: MDS+SC+DP  Use MDS to get articulation invariance signatures.  Build the shape context on the signatures.  Use dynamic programming for shape context matching (a) and (c) show two shapes related by articulation. (b) and (d) show their signatures.

Related work: Shape Contexts for 2D shape The shape context was introduced by Belongie et al. Due to its simplicity and discriminability, the shape context has become quite popular recently in shape matching tasks. It describes the relative spatial distribution.

Related work: Shape Contexts for 2D shape Given n sample on shape. The shape context at points is defined as a histogram of the relative coordinates of the remaining n-1 points. Where the bins uniformly divide the log-polar space. The shape context uses the Euclidean distance to measure the spatial relation between landmark points. This causes less discriminability for complex shapes with articulations.

Inner-Distance Shape Context (IDSC) To extend the shape context, Euclidean distance is directly replaced by the inner- distance.

Inner-Distance Shape Context (IDSC) The angle between the contour tangent at p and the direction of at p is insensitive to articulation, called inner-angle, denoted. Inner-angle is used for the orientation bins. Noise may reduces the stability of the inner- angle, smoothing contour before computing it.

Inner-Distance Shape Context (IDSC) Example: In the histogram, the x axis denotes the orientation bins and the y axis denotes log distance bins.

Shape matching through Dynamic programming Given two shapes A and B, points sequences on their contour, say, for A and for B, assume. A matching from A to B is a mapping. is matched to if, and otherwise left unmatched. should minimize the match cost.

is the penalty for leaving unmatched, and for, is the cost of matching to. and are the shape context histogram of and. K is the number of histogram bins. Shape matching through Dynamic programming

DP is used to solve the matching problem since it uses the ordering information provided by shape contours. By default, assumes the two contours are already aligned at their start and end points. Without this assumption, one simple solution is to try different alignments at all points on the first contour and choose the best one.

Shape matching through Dynamic programming Because shapes can be first rotated according to their moments, it is sufficient to try aligning a fixed number of points, say k points. Usually, k is much smaller than m and n.

Shape distance The matching cost is used to measured the similarity between shapes. IDSC+DP is better than SC+DP  Better performance  Only two parameters to tune The penalty for a point with no matching, usually set 0.3. The number of start points k for different alignments, usually set 4-8.  Easy to implement since it does not require the appearance and transformation model.

Shortest path texture context The combination of texture and shape information, because  Shapes from different classes sometimes are more similar than those from the same class.  Shapes are often damaged due to occlusion and self-overlapping.

Shortest path texture context The texture information along these paths provides a natural articulation insensitive texture description. The angles between intensity gradient directions and shortest path directions are used, called relative orientations.

Shortest path texture context The SPTC for each is a three dimensional histogram.  The inner-distance  The inner-angle  The (weighted) relative orientation The relative orientations are weighted by gradient magnitudes.

Shortest path texture context

Experiment the number of inner-distance bins: or the number of inner-angle bins: the number of relative orientation bins: The number of different starting points for alignment: The penalty for one occlusion:

Experiment (a) Articulated database. (b) MDS of the articulated database using the inner-distance.

Experiment Retrieval result on the articulate data set SC+DPIDSC+DP

Experiment MPEG7 CE-Shape-1 shape database is widely tested, which consists of 1400 silhouette images from 70 classes. Each class has 20 different shapes.  Bullseye test: for every image in the database, it is matched with all other images and the top 40 most similar candidates are counted.

Experiment  The score of the test is the ratio of the number of correct hits of all images to the highest possible number of hits (which is 20x1400).

Experiment The Kimia Database  Data set 1 : 25 instance from six categories

Experiment  Data set 2 : 99 instances from nine categories

Experiment The ETH-80 image set: This data set contains 80 objects from eight classes, with 41 images of each object obtained from different viewpoints.

Experiment: Foliage image retrieval Swedish leaf data set

Experiment: Foliage image retrieval Smithsonian data set: 343 leaf images from 93 species.

Experiment: Foliage image retrieval

Experiment: Human body matching