1. Recognizing Objects in Range Data Using Regional Point Descriptors A. Frome, D. Huber, R. Kolluri, T. Bulow, and J. Malik. Proceedings of the European Conference on Computer Vision, May, 2004.D. Huber a.k.a. 3D Shape Contexts Talk prepared by Nat Duca, duca@jhu.edu

2. Motivation Find instances of known shapes in 2.5D range scans Image source: Frome04

3. 2D Shape Contexts Take a random point on the shape Image source: Belongie02

4. 2D Shape Contexts Compute the offset vectors to all other samples

5. 2D Shape Contexts Histogram the vectors against sectors and shells Perform this for a large sampling of points

6. Extension to 3D Step 1: pick random points on surface Image source: Koertgen03

7. Extension to 3D For each point, compute and histogram offsets Image source: Koertgen03

8. Extension to 3D For each point, compute offsets Image source: Koertgen03

9. Extension to 3D Now we histogram the offset vectors. The 3D histogram of looks like: Image source: Frome04

10. Extension to 3D Shells are spaced logarithmically apart Histogram votes are weighted by the volume of the bin Some L n difference of the histogram vector can be used to compare two contexts Image source: Frome04, Koertgen03

11. Challenges 1. How do we orient the histogram “spheres” 2. How do we compute distance between a model and one of its subsets? 3. Speed

12. Initial histogram orientation Align the object’s north-pole to the surface normal Problems: 1. One degree of freedom remains 2. Histogram values depend on the precision of the surface normals The paper solves both problems using: – Brute force rotation – spherical harmonics

13. Harmonic shape context Each shell’s histogram is a spherical function Convert each shell to a harmonic representation and store the amplitude coefficients only Initial histogram placement doesn’t matter, Noise in surface normals doesn’t affect descriptor Image source: Weisstein04

14. the big picture: Partial Shape Matching For a query shape S q and a stored model S i, their nearness is defined as: A shape context placed randomly on the query surface S q A precomputed shape context for model S i query surface

15. Experiment 1: resilience to noise (a) model with 5cm gaussian noise (b) model with 10cm gaussian noise (c) reference (databased) model Image source: Frome04

16. Experiment 2: partial matching Input: or Output: Image source: Frome04

17. Evaluating the results Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor approach Is their presentation fair?

18. Results for noise Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor Is their presentation fair? Comments: Recognition rate: across 100 trials, how many times did we get the correct answer back the first time? All three techniques are equivalent in absence of noise Image source: Frome04 Results for 5cm noise

19. Results for noise Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor Is their presentation fair? Comments: Why is the harmonic approach doing worse? We expect it to be doing as well or better than the basic approach Image source: Frome04 10cm noise, 55cm normal window

20. Results for noise Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor Is their presentation fair? Comments: Notice how, when the normals are better filtered, the harmonics do better! How can this be so? Image source: Frome04 10cm noise, 105cm normal window

21. Results for partial matching Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor Is their presentation fair? Comments: Rank depth of R means that the correct answer appeared in the top R results. Clearly, the harmonics are throwing away too much Or is the fact that the shells are rotationally independent to blame? Image source: Frome04 View 1

22. Results for partial matching Where does the blame lie: – Spherical histogram – Harmonics representation – Point choice – Representative descriptor Is their presentation fair? Comments: Rank depth of R means that the correct answer appeared in the top R results. The authors claim that the ground is setting off the match Image source: Frome04 View 2

23. Speed considerations We use a spherical hash with J sectors, and KxL latitudinal and longitudinal divisions The basic vector is (roughly) J x K x L in size The harmonic representation is roughly the same size Without harmonics, they must store L extra rotations in order: J x K x L 2 They use Locality Sensitive Hashing to reduce the amount of effort required here:

24. Speed considerations: LSH results Image source: Frome04 Without hashing

25. Summary What was introduced: – 3D histogram extension of 2D shape contexts – A poorly-performing spherical harmonic decomposition of the 3D histogram – The representative decriptor method works pretty well What would have been nice: – Precision of query when the shells are logarithmically or linearly separated – Is the representative descriptor approach the limiting factor? We need more data to confirm or deny!

26. Image sources Frome04: A. Frome, D. Huber, R. Kolluri, T. Bulow, and J. Malik. Proceedings of the European Conference on Computer Vision, May, 2004 Belongie02: S. Belongie et al. Shape matching and object recognition using shape contexts. IEEE Trans on Pattern Analysis and Machine Intelligence. 24(4):509-522, April 2002. Koertgen03: M. Körtgen, G.-J. Park, M. Novotni, R. Klein "3D Shape Matching with 3D Shape Contexts", in proceedings of The 7th Central European Seminar on Computer Graphics, April 2003 Weisstein: Eric W. Weisstein. "Spherical Harmonic." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/SphericalHarmonic.html