Image-based Plant Modeling Zeng Lanling Mar 19, 2008

1.Image-based Plant Modeling 2.Image-based Tree Modeling Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang* The Hong Kong University of Science and Technology * Microsoft Research

Image-based Plant Modeling Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang* The Hong Kong University of Science and Technology * Microsoft Research

Motivation Plants are ubiquitous but difficult to model – Complex geometry and topology – Fine texture details Previous methods have limitations – Manual intensive – Unintuitive – Lack of realism

Features Only a handheld camera is used for capture Ability to capture complex geometry and texture User interaction is small

Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

captured images (35-45 images) cloud of reliable 3D points Image Capture and Structure from Motion Hand-held camera Use quasi-dense approach [Lhuillier & Quan 2005] … …

Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

Leaf Segmentation Goal: Segment 3D points and images into individual leaves Problem: Segmentation is subjective and ill-posed Our solution: Joint segmentation with user interaction

3D segmentation Automatic joint segmentation – Graph model with joint 2D/3D distance – Graph partition Interactive refinement – User interface – Graph update

graph model 3D segmentation —— Construct 3D graph Graph G = { V, E }: V: 3D points recovered from SFM E: each point connected to its K- nearest neighbors

3D segmentation —— Define joint 2D/3D distance Distance between two nodes – 3D distance : 3D Euclidean distance – 2D distance.p.p.q.q pq d 2d (p,q) = gradient of i-th image

3D segmentation —— Graph partition By normalized cut [Shi & Malik 2000] after 3D graph partition initial 3D Graph

2D segmentation By two-label graph-cut algorithm – FG: region covered by projected 3D points in a group – BG: projections of all other points not in the group …… Segmented 2D leaves Clustered 3D points

Interactive refinement Click to confirm segmentation Draw to split and refine Click to merge

Sample session of user interface

3D graph update By two-label graph-cut problem – Min-cut algorithm – Real-time visual feedback before update split stroke after update

Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

Model-based leaf reconstruction Generic leaf extraction Leaf reconstruction – Flat leaf fitting – Boundary warping – Texture extraction – Shape deformation

Generic leaf extraction Extract a flat leaf mesh from image

Flat leaf fitting Estimate position, orientation, and scale by SVD decomposition of each 3D point set

Boundary warping & texturing Match leaf boundary to 2D segmentation boundary using iterative closest point (ICP) algorithm Crop texture after matching leaf boundary segmentation boundary

Shape deformation Move each vertex to the closest 3D point along normal of flat leaf

Overview of system … … 3D2D Image Capture Structure from Motion Leaf Segmentation Leaf Reconstruction Branch Editing Plant Model Render

Interactive Branch Editing Automatic reconstruction is difficult due to significant occlusion We rely on user to: – Add branch – Move branch – Edit branch thickness (through radius) – Specify leaf

Sample session of branch editing

Nephthytis rendering resultmesh modelone source image (1 from 35)

Poinsettia one source image (1 from 35) recovered modelnovel viewpoint

Image-based texture vs. generic texture image-based texturegeneric texture

Schefflera one source image (1 from 40) recovered model

Indoor tree one source image (1 from 45) recovered model

Plant editing recovered modelafter texture replacement Texture replacement

Plant editing original modelafter cut-and-paste Branch cut-and-paste

Reconstruction statistics NephthytisPoinsettiaScheffleraIndoor tree # image # FG pts53,00083,00043,00031,000 # leaves30≈ 120≈ 450≈ 1500 # UAL Recovered leaves BET (min) UAL = user assisted leaves, BET = branch edit time

Conclusions Semi-automatic image-base plant modeling – Simple capturing – Realistic shape and texture Technical contributions: – Interactive joint segmentation – Model-based leaf reconstruction – Interactive branch editing

Future directions Improve joint segmentation Handle more complex plants (e.g., with flowers) Use specialized leaf rendering algorithm

Image-based Tree Modeling Ping Tan, Gang Zeng *, Lu Yuan, Jingdong Wang, Sing Bing Kang, Long Quan The Hong Kong University of Science and Technology * Microsoft Research

Different

Overviwe of the system

Branch recovery Reconstruction of visible branches Graph construction Conversion of sub-graph into branches User interface for branch refinement Reconstruction of occluded branches Unconstrained growth Constrained growth

Visible branches recovery

Occluded branches recovery

Leaves reconstruction Mean shift filtering Region split or merge Color-based clustering User interaction

Mean shift filtering

Leaves reconstruction

Adding leaves to branches Create leaves from segmentation Synthesizing missing leaves

Results

Results

Results

Results

Approaches to plant modeling Rule-based – Geometric rules [Weber&Penn 1995] – L-system [Prusinkiewicz et al. 1994] [Noser et al. 01] – Botanical rules [De Reffye et al. 1988] Image-based – Volumetric [Shlyakhter et al. 2001] [Reche et al. 2004] – Statistical [Han et al. 2003]

Advantages: – Impressive-looking plants, trees, and forests Disadvantages: – Difficult to use for non-expert – Difficult to exactly match appearance of actual plants Rule-based plant modeling [Weber&Penn 1995] [Prusinkiewicz et al. 1994] [Phillippe De Reffye et al. 1988]

Advantages: – Details of real plant are captured in image Disadvantages: – Limited realism (visual hull) – Not manipulable (volumetric representation) Image-based plant modeling [Reche et al. 2004] [Shlyakhter et al. 2001] [Han et al. 2003]

Thanks!