Paper Presentation: Shape and Matching Aashish Thite, Qinyuan Sun, and Dongxi Zheng 11/28/2012

Overview Problems to solve Inter-paper relationship Paper 1 Paper 2 Summary

Problems to Solve Are two objects similar? If yes, how similar are they? Are there similar shapes in distinct pictures?

Problems to Solve How to match two sets of features of logically similar objects efficiently and effectively? (RANSAC is slow and not effective)

Problems to Solve Scene (or object) recognition? How to recognize this scene as an office? How to recognize this animal as a horse?

Overview Problems to solve ✔ Inter-paper relationship Paper 1 Paper 2 Summary

Inter-paper relationship Paper 1 (PAMI 2002): Shape Matching and Object Recognition Using Shape Contexts Paper 3 (ICCV 2005): The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features Paper 4 (CVPR 2006): Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories Paper 2 (CVPR 2007): Matching Local Self-Similarities across Images and Videos

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 Summary

Shape Matching and Object Recognition Using Shape Context By Serge Belongie, Jitendra Malik and Jan Puzicha PAMI 2002

Descriptor: Shape Context

Advantage of Shape Context A rich descriptor Scale invariant Translation invariant Robust under small geometrical distortion Can be made rotation invariant

Cost Between Matching Two Points

Bipartite Graph Matching Minimize Add dummy nodes to each point set with constant matching cost to handle outliers and square the points in two shapes

Modeling Transformation Thin plate spline minimizing the bending energy

Some More Constraints Regularization Parameter Transformation Control the amount of smoothing Transformation

Shape Distance Shape Context Distance Image Appearance Distance Bending Energy

Illustration

Choosing Prototypes What example to store as prototypes? A novel editing algorithm based on shape distance and K-medoids.

Trademark Retrieval Examples

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 ✔ Summary

Matching Local Self-Image Similarity Across Image and Videos by Eli Shechtman and Michal Irani CVPR 2007

Local Self-similarity Descriptor Correlation Surface Image patch 5x5 Surrounding image region 40 pixels varnoise photometric variations varauto maximum variance of the difference of the patches within a small neighborhood

Local Self-similarity Descriptor (cont.) Log-polar histogram 20 angles, 4 radial intervals maximum correlation for each bin normalize to [0..1]

Properties of Descriptor Local descriptor log-polar representation account for local affine deformation Allow for non-rigid deformation Contain rich information No explicit segmentation Handle regions without clear boundary

Global Ensemble of Descriptors Compute local self-similarity densely computed on both F and G 5 pixels apart All Local descriptor in F form ensemble of descriptors

Matching Global Ensembles of Descriptors Filter out non-informative descriptors No local self-similarity High self-similarity (eg. large homogeneous region) Efficient ensemble matching algorithm from another paper Generates a dense likelihood map Done in multi-scale Scale invariant

Applications Objection Detection Image Retrieval by sketching Action Detection in video

Image Retrieval Results

Action Detection Result

Comparison with other descriptor

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 ✔ Summary

Bag of Words http://www.cs.unc.edu/~lazebnik/research/spring08/lec17_bag_of_features.ppt

http://www.cs.unc.edu/~lazebnik/research/spring08/lec17_bag_of_features.ppt

Image Representation

Summary: Pyramid match kernel • Linear time complexity • Insensitive to clutter • Positive-definite function

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 ✔ Summary

Svetlana Lazebnik, Cordelia Schmid, Jean Ponce, 2006 Many of the Slides by Svetlana Lazebnik: www.cs.illinois.edu/homes/slazebni/slides/ima_poster.pdf

Key Idea •Pyramid Match Kernel (Grauman & Darrell) Pyramid in feature space, ignore location •Spatial Pyramid (this work) Pyramid in image space, quantize features

Quantize Feature Space

Spatial Histograms

Spatial Histograms

Spatial Matching Kernel Train an SVM using this Kernel

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 ✔ Summary

Summary An effective descriptor for features around a shape should be informative, e.g., shape context (paper 1), self-similarity descriptor (paper 2). For shape or scene recognition, the features in two images typically have no 1-on-1 mapping relation, not even the same amount. Multi-level histograms (paper 3 and 4) are an efficient and effective matching approach.

Overview Problems to solve ✔ Inter-paper relationship ✔ Paper 1 ✔ Summary ✔ Questions

Thank You!