1. Image-based Plant Modeling Zeng Lanling Mar 19, 2008

2. 1.Image-based Plant Modeling 2.Image-based Tree Modeling Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang* The Hong Kong University of Science and Technology * Microsoft Research

3. Image-based Plant Modeling Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang* The Hong Kong University of Science and Technology * Microsoft Research

4. Motivation Plants are ubiquitous but difficult to model – Complex geometry and topology – Fine texture details Previous methods have limitations – Manual intensive – Unintuitive – Lack of realism

5. Features Only a handheld camera is used for capture Ability to capture complex geometry and texture User interaction is small

6. Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

7. Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

8. captured images (35-45 images) cloud of reliable 3D points Image Capture and Structure from Motion Hand-held camera Use quasi-dense approach [Lhuillier & Quan 2005] … …

9. Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

10. Leaf Segmentation Goal: Segment 3D points and images into individual leaves Problem: Segmentation is subjective and ill-posed Our solution: Joint segmentation with user interaction

11. 3D segmentation Automatic joint segmentation – Graph model with joint 2D/3D distance – Graph partition Interactive refinement – User interface – Graph update

12. graph model 3D segmentation —— Construct 3D graph Graph G = { V, E }: V: 3D points recovered from SFM E: each point connected to its K- nearest neighbors

13. 3D segmentation —— Define joint 2D/3D distance Distance between two nodes – 3D distance : 3D Euclidean distance – 2D distance.p.p.q.q pq d 2d (p,q) = gradient of i-th image

14. 3D segmentation —— Graph partition By normalized cut [Shi & Malik 2000] after 3D graph partition initial 3D Graph

15. 2D segmentation By two-label graph-cut algorithm – FG: region covered by projected 3D points in a group – BG: projections of all other points not in the group …… Segmented 2D leaves Clustered 3D points

16. Interactive refinement Click to confirm segmentation Draw to split and refine Click to merge

17. Sample session of user interface

18. 3D graph update By two-label graph-cut problem – Min-cut algorithm – Real-time visual feedback before update split stroke after update

19. Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

20. Model-based leaf reconstruction Generic leaf extraction Leaf reconstruction – Flat leaf fitting – Boundary warping – Texture extraction – Shape deformation

21. Generic leaf extraction Extract a flat leaf mesh from image

22. Flat leaf fitting Estimate position, orientation, and scale by SVD decomposition of each 3D point set

23. Boundary warping & texturing Match leaf boundary to 2D segmentation boundary using iterative closest point (ICP) algorithm Crop texture after matching leaf boundary segmentation boundary

24. Shape deformation Move each vertex to the closest 3D point along normal of flat leaf

25. Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

26. Interactive Branch Editing Automatic reconstruction is difficult due to significant occlusion We rely on user to: – Add branch – Move branch – Edit branch thickness (through radius) – Specify leaf

27. Sample session of branch editing

28. Nephthytis rendering resultmesh modelone source image (1 from 35)

29. Poinsettia one source image (1 from 35) recovered modelnovel viewpoint

30. Image-based texture vs. generic texture image-based texturegeneric texture

31. Schefflera one source image (1 from 40) recovered model

32. Indoor tree one source image (1 from 45) recovered model

33. Plant editing recovered modelafter texture replacement Texture replacement

34. Plant editing original modelafter cut-and-paste Branch cut-and-paste

35. Reconstruction statistics NephthytisPoinsettiaScheffleraIndoor tree # image35 4045 # FG pts53,00083,00043,00031,000 # leaves30≈ 120≈ 450≈ 1500 # UAL6216935 Recovered leaves291163741036 BET (min)521540 UAL = user assisted leaves, BET = branch edit time

36. Conclusions Semi-automatic image-base plant modeling – Simple capturing – Realistic shape and texture Technical contributions: – Interactive joint segmentation – Model-based leaf reconstruction – Interactive branch editing

37. Future directions Improve joint segmentation Handle more complex plants (e.g., with flowers) Use specialized leaf rendering algorithm

38. Image-based Tree Modeling Ping Tan, Gang Zeng *, Lu Yuan, Jingdong Wang, Sing Bing Kang, Long Quan The Hong Kong University of Science and Technology * Microsoft Research

39. Different

40. Overviwe of the system

41. Branch recovery Reconstruction of visible branches Graph construction Conversion of sub-graph into branches User interface for branch refinement Reconstruction of occluded branches Unconstrained growth Constrained growth

42. Visible branches recovery

43. Occluded branches recovery

44. Leaves reconstruction Mean shift filtering Region split or merge Color-based clustering User interaction

45. Mean shift filtering

46. Leaves reconstruction

47. Adding leaves to branches Create leaves from segmentation Synthesizing missing leaves

48. Results

49. Results

50. Results

51. Results

52. Approaches to plant modeling Rule-based – Geometric rules [Weber&Penn 1995] – L-system [Prusinkiewicz et al. 1994] [Noser et al. 01] – Botanical rules [De Reffye et al. 1988] Image-based – Volumetric [Shlyakhter et al. 2001] [Reche et al. 2004] – Statistical [Han et al. 2003]

53. Advantages: – Impressive-looking plants, trees, and forests Disadvantages: – Difficult to use for non-expert – Difficult to exactly match appearance of actual plants Rule-based plant modeling [Weber&Penn 1995] [Prusinkiewicz et al. 1994] [Phillippe De Reffye et al. 1988]

54. Advantages: – Details of real plant are captured in image Disadvantages: – Limited realism (visual hull) – Not manipulable (volumetric representation) Image-based plant modeling [Reche et al. 2004] [Shlyakhter et al. 2001] [Han et al. 2003]

55. Thanks!