Symmetric Architecture Modeling with a Single Image Author: Nianjuan Jiang, Ping Tan, Loong-Fah Cheong Department of Electrical & Computer Engineering, National University of Singapore Presenter: Feilong Yan

Motivation Model architecture from single image is common task in 3D creation due to the lack of the more images. Historic Photo: Internet Photo:

Motivation Single image based modeling is very difficult! Due to the trouble on camera calibration and texture loss The recent methods only can handle simple and planar façade Pascal Mulle et al. Image-based procedural modeling of facades Changchang Wu et al. Repetition-based Dense Single-View Reconstruction

Motivation But what about this one? And If we only have this single photo. Complex and not planar

Motivation Fortunately , the symmetry is very prevalent in the architecture Symmetry is a breakthrough which magically can generate more images from the input Complex and not planar This is reasonable, but exciting to me

Main Idea Bilateral Symmetry

Main Idea Rotational Symmetry

2 even more views Reconstruction Main Idea 2 even more views Reconstruction

Modeling Pipeline 3D Reconstruction Surface Modeling Texture Enhancement Input Image and Frustum Vertices Model Refinement Model Initialization Calibration and 3D Reconstruction

3D points Reconstruction 3D Reconstruction Camera Calibration 3D points Reconstruction

Camera Calibration Previous Methods Calibrate the camera from the vanishing points of 3 mutually orthogonal directions in a single image. HARTLEY, R., AND ZISSERMAN Multiple View Geometry in Computer Vision But many photos do not have 3 vanishing points, and this method is often numerical unstable

Camera Calibration Previous Methods If enough(>=6) correspondences between spatial vertices and the image pixels are known, the camera calibration may be immediately computed. WILCZKOWIAK, M. et. al Using geometric constraints through parallelepipeds for calibration and 3d modeling Parallelipiped is used to represent a building block. Under the constraint of parallelipiped, the visible 6 spatial vertices may be estimated. This method is stable but not very suitable for some architecture

Camera Calibration New Method: Inspired by parallelipiped method, the author found the frustum more general to represent the architecture

Camera Calibration Demo: Frustum is symmetric

Camera Calibration 6 4 5 3 1 2 Coordinate represented in world: 𝑃 𝑖 =𝛬⋅ 𝑃 𝑖 𝑃 𝑖 =(𝑥 𝑖 , 𝑦 𝑖 , 𝑧 𝑖 𝑥 𝑖 , 𝑦 𝑖 ∈{1,−1 𝑧 𝑖 ∈{0,1 𝛬= 𝑙 1     𝑙 2 cos𝜃     0      0 0    𝑙 2 sin𝜃      0      0 0       0        𝑙 3 𝛼      0 0       0       1 𝛼        1 𝛬= 𝑙 1    0     0      0 0   𝑙 2       0      0 0       0        𝑙 3 𝛼      0 0       0       1 𝛼        1 Of this example

Camera Calibration 6 4 5 3 1 2 𝑝 𝑖 =( 𝑢 𝑖 , 𝑣 𝑖 , 𝑤 𝑖 𝑝 𝑖 =( 𝑢 𝑖 ,  𝑣 𝑖 ,  𝑤 𝑖 K= 𝑓 𝛼 𝑥 𝑠 𝑝 𝑥 0 𝑓 𝛼 𝑦 𝑝 𝑦 0 1 𝑝 𝑖 ≃𝑀 𝑃 𝑖 =𝑀𝛬 𝑃 𝑖 = 𝑀 𝑃 𝑖 𝑀 = K⋅ R t ⋅𝛬 R= Quaternion( unit vector(x,y,z),𝜃) M=K⋅ R t (3∗4 matrix) t =t(x, y, z) 15 parameters to estmate 𝑙 3 =1 𝑙 1 𝑙 2 𝜃 𝛼

Camera Calibration 6 4 5 3 1 2 Simplification: K= 𝑓 0 0 𝑓 0 1 11 parameters to estimate, now the calibration is formulated as a non-linear optimization

Camera Calibration Optimization Initialization: 𝑀 = K⋅ R t ⋅𝛬 The Quadratic : 𝐾 −1 𝑀 𝑇 ⋅( 𝐾 −1 𝑀 )= [𝑅|𝑡]⋅𝛬 𝑇 ⋅([𝑅|𝑡]⋅𝛬 Extend the right multiplication, since the R is unit orthogonal matrix, then we obtain: 𝑚 1 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 1 = 𝑙 1 2 𝑚 1 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 2 = 𝑙 1 𝑙 2 cos𝜃 User gives the 𝜃 𝑚 2 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 2 = 𝑙 2 2

3D Points Reconstruction Symmetry-Based Triangulation:

3D Points Reconstruction Symmetry-Based Triangulation:

Surface Modeling User-Interaction Assisted Modeling Geometry Modeling Model Refinement Texture Mapping

Geometry Modeling Roof Planar Structure

Model Refinement

Texture Mapping Single image inevitably lack texture samples due to the foreshortening and occlusion. But to achieve a good texture effect, there are 2 requirements: 1. the final texture should be consistent with the foreshortened image ; 2, the final texture should have consistent weathering pattern. We need to know where is well textured and where not Refine low quality region Detect Texture Quality Texture the occluded region

Texture Quality Detection Back Project Ratio = Triangle.size / imageProjection.size Ratio > Threshold and Ratio is finite: large texture distortion Ratio<Threshold: distortion free Ratio is infinite: occluded Texture in distortion free region will be used as the texture sample

Refinement for Low-Quality Super- Resolution Problem

Occluded Region Texturing The simplest way is to repeat the same texture as those of their symmetric counterparts, but this makes the model look artificial. It is better to synthesize the texture in these region with common method Another feature of the texture is weathering pattern, a constraint texture map is used according to the height of the architecture. Input sample Synthesized

Result

Result

Conclusion Contribution: Limitations: Novel Calibration Method Texture Enhance Method Limitations: Strong assumption for simplification of camera calibration

Message from this Paper