1. Indexing with Unknown Illumination and Pose Ira Kemelmacher Ronen Basri Weizmann Institute of Science

2. The task – Shape Indexing Came l Hippo Dino 2. Recover pose+lighting 1. Recognize …

3. Why is it hard? Unknown pose Unknown lighting Clutter Occlusion

4. Assumptions Weak perspective projection 3D rigid transformation Lambertian model

5. Previous… Identification using alignment Fischler and Bolles, Huttenlocher and Ullman “3D to 2D invariants do not exist” Burns etal., Moses etal., Clemens etal. Indexing faster than alignment Jacobs, Wolfson etal.

6. Previous Indexing Methods Ignored intensity information Need many point or line features Restricted to polyhedral objects

7. Our algorithm… Handles both pose and lighting Uses intensities to filter out incorrect matches Still relies on point features but only very few are needed General objects

8. Indexing with pose - Affine model (Jacobs ‘96) p i = AP i + t  8 DOF -> 5 points (n 1,n 2,m) (α 3, β 3 ) (α 4, β 4 ) p0p0 p1p1 p2p2 p3p3 p4p4 P0P0 P1P1 P3P3 P4P4 P2P2

9. (n 1,n 2,m) P0P0 P1P1 P3P3 P4P4 P2P2 Representation in two 2D tables Online – matching Offline – preprocessing (α 3, β 3 ) (α 4, β 4 ) p0p0 p1p1 p2p2 p3p3 p4p4 α -space α 4 = α 3 m + n 1 (α 3, α 4 ) β -space β 4 = β 3 m + n 2 (β 3, β 4 )

10. Modifications – still two 2D spaces Online – matching Offline – preprocessing (α 3, β 3 ) (α 4, β 4 ) p0p0 p1p1 p2p2 p3p3 p4p4 α –dual-space (n 1,n 2,m) P0P0 P1P1 P3P3 P4P4 P2P2 r 1 = α 3 cos θ+ α 4 sinθ (θ,r 1 ) β –dual-space r 2 = β 3 cos θ+ β 4 sinθ (θ,r 2 )

11. One 3D space Online – matching Offline – preprocessing (α 3, β 3 ) (α 4, β 4 ) p0p0 p1p1 p2p2 p3p3 p4p4 (n 1,n 2,m) P0P0 P1P1 P3P3 P4P4 P2P2 (θ,r 1,r 2 )

12. False Matches True match

13. How to eliminate the false matches Enforce rigidity using inverse Gramian Test - Weinshall, ‘93 Consistency with lighting  NEXT

14. Harmonic Images – Linear Basis for Lighting (Basri and Jacobs ‘01, Ramamoorthi and Hanrahan ‘01) 

15. Representation by harmonics I = b * H Harmonics of model point set Unknown light Intensitie s of image point set

16. The consistency measure For corresponding image and model sets this is minimal I = b * H Should we apply it on feature points?

17. “Smooth points” Smooth points Feature points

18. Voting Sets of points that pass the lighting test vote for their respective model All models receive scores: Score = fraction of image sets for which the model appears min Once model is selected its corresponding subsets used to determine its pose and lighting

19. Experiments Real 3d objects acquired using laser scanner Feature points collected automatically using Harris corner detector

20. Results dino hippo camel shark pinokio bear elephant face

21. Results dinohippocamel shark pinokio bear elephant face

22. Results

23. Results – Indoor scene dinohippocamel shark pinokio bear elephant face

24. Results – Outdoor scene dino camel shark pinokio bear elephant face hippo

25. Results – Night Scene dino camel shark pinokio bear elephant face hippo

26. Results – Night Scene 2 dino camel shark pinokio bear elephant face hippo

27. Filtering out matches

28. How much does lighting help? Voting based on Affine model

29. How much does lighting help? Affine + Rigidity test

30. How much does lighting help? Affine + Rigidity + Lighting

31. Conclusion Identify 3d objects in 2d scenes Unknown pose, light Clutter, occlusions General, real objects Fast, efficient Combination of intensity cues and geometry

32. Thank you!