1. Interactive Point-based Modeling of Complex Objects from Images Pierre Poulin (a,b) Marc Stamminger (a,c) François Duranleau (b) Marie-Claude Frasson (a) George Drettakis (a) (a) REVES, INRIA Sophia Antipolis (b) DIRO, Université de Montréal (c) University of Erlangen

2. Modeling Complex Objects

3. High visual complexity Time consuming Algorithms for specialized objects –e.g., plants, mountains, etc. Adaptive rendering Many applications need such objects

4. Key Observations Extracting complex models from photos is a very powerful approach Point-based representation is very effective for complex models –Efficient display and storage User interaction is beneficial when extracting quality models –Specify where details are needed –Resolve some ambiguities

5. Image-based Point Modeling Images are very flexible –Reality-based (photos) –Acquisition is easy

6. Image-based Point Modeling Points are very flexible –Fast rendering (hardware support) –Adaptive rendering for interactive display Stamminger

7. Image-based Point Modeling Points are very flexible –Hierarchical organization and levels of detail Q-splat

8. Image-based Point Modeling Points are very flexible –Visual quality –Many recent advances Deussen

9. Automatic Reconstruction Images Reconstruction Process Constraints 3D Model Image

10. Interactive Reconstruction Images Reconstruction Process Constraints 3D Model Image User new images requantize recalibrate

11. Interactive Reconstruction Images Reconstruction Process Constraints 3D Model Image User color comparisons plausibility threshold new depth maps zone of interest

12. Interactive Reconstruction Images Reconstruction Process Constraints 3D Model Image User revalidate the points request more points decimate the points jitter the points sample with patterns hole filling

13. Interactive Reconstruction Images Reconstruction Process Constraints 3D Model Image User undo changes remove points add polygons

14. Interactive Reconstruction Interactive display –6 M points/sec. on a PIII 1GHz with GeForce3 Efficient reconstruction algorithm –Test more than 1K points/sec. Simple and intuitive controls –Direct interaction with the points

15. Computer Vision Contributions 3D scanners Structured light Stereo – N-views Shape-from-X Volumetric

16. Volumetric Reconstruction Voxel coloring and Space carving –If a voxel is impossible, carved out of object –Silhouettes, transparency, shading –Photo-consistency SeitzKutulakos

17. Image-based Polygon Modeling Academic: Façade, Rekon, Reality Industry: RealViz, Canoma, Photomodeler Façade

18. Image-based Polygon Modeling Small polygonal scene (30-100 polygons) Extracted textures and illumination Boivin

20. Input Images (4/14)

21. Input Images Digital camera: Canon EOS-DS30 1080x720 and 2166x1440 Fixed aperture and shutter speed Try not to change zoom OpenGL and ray traced test scenes

22. Camera Calibration

23. ImageModeler from RealViz Fiduciary marks placed around the object Interactive system Intrinsic and extrinsic camera parameters

24. 3D Zone of Interest

25. Initial Random Points

26. Generated randomly within the envelope More specific patterns discussed later Projection of a point in each photo Gather colors

27. Color Comparison Euclidean distance –RGB, CIE xy, CIE Luv, CIE Lab –Speed vs. accuracy Color quantized images –Precomputed (ppmquantall or more sophisticated) –Quantization only on projected zone of interest –32 to 128 colors –Reduce shading variations –Efficient test for color equality

28. C: 25% B: 50% Dominant Color A: 100% Plausibility 100% 33% with visibility

29. Random Points with Depth Maps

30. Depth Maps Computed from the current set of points Updated on user demand With depth maps, can raise the plausibility threshold Generate more points within the object Re-evaluation of previously generated points

31. Clean-up Points

32. In general –Increase color threshold and re-evaluate With good depth maps –Project in each image –Reject if point visible and color too different

33. Generate More Points

34. Randomly Stratified sampling based on voxels Point decimation based on voxels

35. Guide the Points

36. Smaller 3D sphere of interest –Generate more points –Eliminate all points 3D flood fill for branching patterns Patterns for planar surfaces Patterns for boundary surfaces

37. Filling with no Leaves

38. Filling with Leaves

39. Jitter the Points

40. Reprojection

41. Stepping through it again

42. Results SceneImagesResolutionColorsPoints Fruit bowl13512x512-- Soldier132160x144064118K Snack81440x96064120K Ficcus132160x144064150K

43. Synthetic Fruit Bowl ray tracingcolor pointsreprojection

44. Toy Soldier color points reprojection

45. Snack

47. Ficcus

48. Conclusions Point-based reconstruction of complex objects from images Tight integration –3D color point representation –User-driven and/or automatic reconstructions –Interactive display Flexible to integrate most advances in computer vision

49. Findings First steps are encouraging, but objects are still of limited realism Information in photos is inspiring, but also difficult to analyse correctly How many things in a pixel? How many pixels and colors for an object?

50. Future Work Video sequences High dynamic range photos Shadows and shading in color comparison Extraction of limited BRDFs 3D texture synthesis of materials

51. Questions Did you… Is it… Can you… When… What… Where…

52. User Interaction in Modeling Specify regions of interest, thresholds, validity Control the visual quality Iterative refining process Guide the solution Automatic or interactive process Interactive display (6 M points/sec. GeForce3)

53. Image-based Point Modeling Difficulties with points –Visibility Holes in surfaces, size of points Filtering the representation and the texture Not our goal to fix these difficulties for now

54. LOD in Graphics Environment maps Billboards Textured polygons Layer-depth images Light field / lumigraph

55. 3D Scanners Very good results in general Size of the scanner wrt object Costs Fixed illumination

56. Stereo - N views Camera calibration Epipolar constraints Color matching 3D position and color Difficulties –Holes and occlusions –Sharp edges, noise, shading Infinity of shapes in general Targeted for object recognition and collision avoidance Only recently goal of photo-realism

57. Shape-from-X Silhouettes Shadows Focus/defocus Motion Shading etc.