Presenter : Jia-Hao Syu 4/26/2017 Multi-View Reconstruction Preserving Weakly-Supported Surfaces (CVPR 2011) M. Jancosek and T. Pajdla Czech Technical University in Prague multi-view Weakly-supported surfaces : low textured walls, windows, cars and ground planes preserve small points in that surface Czech Technical University Presenter : Jia-Hao Syu 4/26/2017

Motivation 4/26/2017 Original image 3D cloud point (special camera offer depth information of every pixel: introduce later) [15] : introduce later , that bottle can not reconstruct well Author’s method 4/26/2017

Outline Related Work[15] Weakly-Supported Surfaces Idea 4/26/2017 Outline Related Work[15] System diagram Weakly-Supported Surfaces Idea Modified weights Results Conclusion This is outline 4/26/2017

4/26/2017 Related Work [15] P. Labatut, J. Pons and R.Keriven, “Robust and efficient surface reconstruction from range data”, In Computer Graphics Forum, 2009 2009 計算機圖形論壇 Target Target : Reconstruct a surface from a set of merged scans (noisy and outliers) 4/26/2017

System Diagram 3D cloud points for each cameras 4/26/2017 System Diagram 3D cloud points for each cameras Combine to one 3D cloud points Delaunay tetrahedralization of a cloud point Surface Reconstruction by graph-cut method 3D Delaunay Triangulation Many image with 3D cloud points Combine to one 3D cloud points Delaunay tetrahedralization Surface Reconstruction by graph cut Later , I will introduce each system block diagram. 4/26/2017

System Diagram 3D cloud points for each cameras 4/26/2017 System Diagram 3D cloud points for each cameras Combine to one 3D cloud points Delaunay tetrahedralization of a cloud point Surface Reconstruction by graph-cut method 3D Delaunay Triangulation Many image with 3D cloud points Combine to one 3D cloud points Delaunay tetrahedralization Surface Reconstruction by graph cut Later , I will introduce each system block diagram. 4/26/2017

3D Scanning Technique Contact 4/26/2017 3D Scanning Technique Contact Non-Contact Time-of-flight camera : a range imaging camera system that resolves distance based on the known speed of light Contact object surface Non-contact object surface One method is to use Time of flight camera A transmitter and a sensor Compute time for round-trip between A and B D : distance c : speed of light t :  time for round-trip between A and B 4/26/2017

System Diagram 3D cloud points for each cameras 4/26/2017 System Diagram 3D cloud points for each cameras Combine to one 3D cloud points Delaunay tetrahedralization of a cloud point Surface Reconstruction by graph-cut method 3D Delaunay Triangulation Many image with 3D cloud points Combine to one 3D cloud points Delaunay tetrahedralization Surface Reconstruction by graph cut Later , I will introduce each system block diagram. 4/26/2017

4/26/2017 3D Cloud Points Acquire a depth map of each camera by 3D scanning technique Compute depth maps of a 3D cloud points to every camera by plane-sweeping method We have many depth maps image Compute a 3D cloud points by plane-sweeping method 4/26/2017

Plane-sweeping Method 4/26/2017 Plane-sweeping Method Pick one pixel P with depth d d 1. Choose one reference image and we pick one pixel p with depth d P Reference Image 4/26/2017

Plane-sweeping Method 4/26/2017 Plane-sweeping Method Find the nearest n target cameras(ex. n = 4) d P Reference Image 4/26/2017

Plane-sweeping Method 4/26/2017 Plane-sweeping Method Compute photo consistency by normalized cross-correlation(NCC) d 5*5 windows Related Camera P Target Image Reference Image 4/26/2017

Plane-sweeping Method 4/26/2017 Plane-sweeping Method Normalized Cross-Correlation The formulation is from wiki n : the pixel number f(x,y) : reference image t(x,y) : target image The value of NCC is between -1 and 1  cross-correlation is a measure of similarity of two signal as a function 光源不同 使亮度不均勻 使用Normalized來處理 Set one threshold to preserve the pixel 4/26/2017

4/26/2017 One 3D Cloud Points Compute photo-consistence between reference and target image One 3D cloud points can be built Get all depth-maps of each camera by using a related camera matrix By the method which is mention before , we can get the one 3D cloud points Delaunay triangulation is common used method for surface reconstruction 4/26/2017

System Diagram 3D cloud points for each cameras 4/26/2017 System Diagram 3D cloud points for each cameras Combine to one 3D cloud points Delaunay tetrahedralization of a cloud point Surface Reconstruction by graph-cut method 3D Delaunay Triangulation Many image with 3D cloud points Combine to one 3D cloud points Delaunay tetrahedralization Surface Reconstruction by graph cut Later , I will introduce each system block diagram. 4/26/2017

2D Delaunay Triangulation Three points can draw a triangle Add one more point or 4/26/2017

2D Delaunay Triangulation Draw a circumcircle of triangle or 4/26/2017

2D Delaunay Triangulation 4/26/2017 2D Delaunay Triangulation Give a set of P point in 2D 4/26/2017

2D Delaunay Triangulation 4/26/2017 2D Delaunay Triangulation No point in P is inside the circumcircle of any triangle 3D Delaunay triangulation : no point in P is inside the circumsphere of any tetrahedralization 4/26/2017

3D Delaunay triangulation Example for 3D Delaunay triangulation 4/26/2017

System Diagram 3D cloud points for each cameras 4/26/2017 System Diagram 3D cloud points for each cameras Combine to one 3D cloud points Delaunay tetrahedralization of a cloud point Surface Reconstruction by graph-cut method 3D Delaunay Triangulation Many image with 3D cloud points Combine to one 3D cloud points Delaunay tetrahedralization Surface Reconstruction by graph cut Later , I will introduce each system block diagram. 4/26/2017

Surface Reconstruction 4/26/2017 Surface Reconstruction We build the 3D Delaunay triangulation How do you reconstruct surface of the object? Concept : 3D cloud points are dense near the object surface (cost is small) S-t graph cut algorithm Maybe with some noise points, the 3D cloud points are almost dense near the object surface We want to build the surface by these 3D cloud points and 3D Delaunay triangulation 4/26/2017

4/26/2017 S-t Graph Cut Source and sink become separated the node of set by a cut the cost of a cut : Minimum cut : a cut whose cost is the least over all cuts We want to use s-t Graph cut but our information are 3D points and Delaunay triangulation 4/26/2017

Define Parameters Node : Delaunay tetrahedralization Edge : triangulation between adjacent tetrahedralizations s(source) : outside of the surface t(sink) : inside of the surface P 4/26/2017

S-t Graph Cut Algorithm Perform a Delaunay Triangulation of the 3d point cloud 4/26/2017

S-t Graph Cut Algorithm Add a node P from left tetrahedralization P 4/26/2017

S-t Graph Cut Algorithm Add a node Q from right tetrahedralization P Q 4/26/2017

S-t Graph Cut Algorithm Add two s and t nodes s P Q t 4/26/2017

S-t Graph Cut Algorithm 10 P Q 3 10 t 4/26/2017

S-t Graph Cut Algorithm This is the surface we want Outside the surface inside the surface 4/26/2017

Assigned Weight 3D cloud points to camera center(line of sight) 4/26/2017 Assigned Weight 3D cloud points to camera center(line of sight)     Sigma越大 surface可以選的路線越多 smooth效果上升 4/26/2017

4/26/2017

Formulation of Cost Function : Visibility Information from points, cameras : Quality of reconstructed surface in terms of size of triangles 4/26/2017

Weakly Supported Surfaces Not photo consistent surface : Low-textured walls, windows, cars and ground planes

Idea Other information to reconstruct weakly supported surface 4/26/2017 Idea Other information to reconstruct weakly supported surface Visual Hull Camera watch information including algorithm Union of foreground image Increasing of number of camera 4/26/2017

Idea Define free-support-space 4/26/2017 Idea Define free-support-space Highly-supported-free space : union of dense 3D points Weakly-supported surface with weakly sampled by 3D points are close to the highly-supported-free space Increasing of camera , the highly-supported-free space increase 4/26/2017

Free-space-support T pi r pj Original 3D cloud points 4/26/2017 Free-space-support T pi r pj Original 3D cloud points Noise with small alpha vision value because seldom points around the noise point X : 3D cloud points(before photo-consistence) 4/26/2017

Target Large Jump in Free Space Support as we go from outside to inside. Next, I give a example of weight assumption

Old T-weights

Modified Weights 4/26/2017 x = w56-w78

Setting up t-weight 4/26/2017

System and Spend Time System OS : 64-bit Win7 CPU : Inter Core i7 RAM : 12GB Dataset Castle data : 30 images with 3072*2048 resolution Dragon data : 114 images with 1936*1296 resolution DataSet/Method Baseline[CFG 09](mins) Ours(mins) Castle 30 32 Dragon 90 94

Results INPUT IMAGE POINT CLOUD CFG-09 OUR METHOD

Results INPUT IMAGE POINT CLOUD CGF-09 OUR METHOD

Demo Video Images : http://www.youtube.com/watch?v=uwluzq5LUn0&feature=player_embedded Demo video http://www.youtube.com/watch?v=UgkB7ITpNaE&feature=player_embedded 4/26/2017

Conclusion Resolve weakly-supported surface by using the information of free-support-space 4/26/2017

Reference M. Jancosek and T. Pajdla, “Multi-View Reconstruction Preserving Weakly-Supported Surface”,  IEEE Conference on Computer Vision and Pattern Recognition, 2011 P. Labatut, J. Pons and R.Keriven, “Robust and efficient surface reconstruction from range data”, In Computer Graphics Forum, 2009 P. Labatut, J. Pons, and R. Keriven., “Efficient Multi-view Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts”, International Conference on Computer Vision, 2007 4/26/2017

Reference M. Jancosek and T. Pajdla , ”Hallucination-free multi-view stereo.”, In RMLE , 2010 M. Jancosek and T. Pajdla, “Removing hallucinations from 3D reconstructions”, Technical Report CMP CTU, 2011 4/26/2017