1. Symmetric Architecture Modeling with a Single Image
Author: Nianjuan Jiang, Ping Tan, Loong-Fah Cheong
Department of Electrical & Computer Engineering, National University of Singapore
Presenter: Feilong Yan

2. Motivation
Model architecture from single image is common task in 3D creation due to the lack of the more images.
Historic Photo:
Internet Photo:

3. Motivation Single image based modeling is very difficult!
Due to the trouble on camera calibration and texture loss
The recent methods only can handle simple and planar façade
Pascal Mulle et al. Image-based procedural modeling of facades
Changchang Wu et al. Repetition-based Dense Single-View Reconstruction

4. Motivation But what about this one?
And If we only have this single photo.
Complex and not planar

5. Motivation
Fortunately , the symmetry is very prevalent in the architecture
Symmetry is a breakthrough which magically can generate
more images from the input
Complex and not planar
This is reasonable, but exciting to me

6. Main Idea
Bilateral Symmetry

7. Main Idea
Rotational Symmetry

8. 2 even more views Reconstruction
Main Idea
2 even more views Reconstruction

9. Modeling Pipeline 3D Reconstruction Surface Modeling
Texture Enhancement
Input Image and Frustum Vertices
Model Refinement
Model Initialization
Calibration and 3D Reconstruction

10. 3D points Reconstruction
3D Reconstruction
Camera Calibration
3D points Reconstruction

11. Camera Calibration Previous Methods
Calibrate the camera from the vanishing points of 3 mutually orthogonal directions in a single image.
HARTLEY, R., AND ZISSERMAN
Multiple View Geometry in Computer Vision
But many photos do not have 3 vanishing points, and this method is often numerical unstable

12. Camera Calibration Previous Methods
If enough(>=6) correspondences between spatial vertices and the image pixels are known, the camera calibration may be immediately computed.
WILCZKOWIAK, M. et. al Using geometric constraints through parallelepipeds for calibration and 3d modeling
Parallelipiped is used to represent a building block. Under the constraint of parallelipiped, the visible 6 spatial vertices may be estimated.
This method is stable but not very suitable for some architecture

13. Camera Calibration New Method:
Inspired by parallelipiped method, the author found the frustum more general to represent the architecture

14. Camera Calibration
Demo:
Frustum is symmetric

15. Camera Calibration 6 4 5 3 1 2 Coordinate represented in world:
𝑃 𝑖 =𝛬⋅ 𝑃 𝑖
𝑃 𝑖 =(𝑥 𝑖 , 𝑦 𝑖 , 𝑧 𝑖
𝑥 𝑖 , 𝑦 𝑖 ∈{1,−1
𝑧 𝑖 ∈{0,1
𝛬= 𝑙 1     𝑙 2 cos𝜃     0      0 0    𝑙 2 sin𝜃      0      0 0       0        𝑙 3 𝛼      0 0       0       1 𝛼        1
𝛬= 𝑙 1    0     0      0 0   𝑙 2       0      0 0       0        𝑙 3 𝛼      0 0       0       1 𝛼        1
Of this example

16. Camera Calibration 6 4 5 3 1 2 𝑝 𝑖 =( 𝑢 𝑖 , 𝑣 𝑖 , 𝑤 𝑖
𝑝 𝑖 =( 𝑢 𝑖 ,  𝑣 𝑖 ,  𝑤 𝑖
K= 𝑓 𝛼 𝑥 𝑠 𝑝 𝑥 0 𝑓 𝛼 𝑦 𝑝 𝑦
𝑝 𝑖 ≃𝑀 𝑃 𝑖 =𝑀𝛬 𝑃 𝑖 = 𝑀 𝑃 𝑖
𝑀 = K⋅ R t ⋅𝛬
R= Quaternion( unit vector(x,y,z),𝜃)
M=K⋅ R t (3∗4 matrix)
t =t(x, y, z)
15 parameters to estmate
𝑙 3 =1
𝑙 𝑙 2 𝜃 𝛼

17. Camera Calibration 6 4 5 3 1 2 Simplification: K= 𝑓 0 0 𝑓 0 1
11 parameters to estimate, now the calibration is formulated as a non-linear optimization

18. Camera Calibration Optimization Initialization: 𝑀 = K⋅ R t ⋅𝛬
The Quadratic :
𝐾 −1 𝑀 𝑇 ⋅( 𝐾 −1 𝑀 )= [𝑅|𝑡]⋅𝛬 𝑇 ⋅([𝑅|𝑡]⋅𝛬
Extend the right multiplication, since the R is unit orthogonal matrix, then we obtain:
𝑚 1 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 1 = 𝑙 1 2
𝑚 1 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 2 = 𝑙 1 𝑙 2 cos𝜃
User gives the 𝜃
𝑚 2 𝑇 ( 𝐾 −𝑇 ⋅ 𝐾 −1 ) 𝑚 2 = 𝑙 2 2

19. 3D Points Reconstruction
Symmetry-Based Triangulation:

20. 3D Points Reconstruction
Symmetry-Based Triangulation:

21. Surface Modeling User-Interaction Assisted Modeling Geometry Modeling
Model Refinement
Texture Mapping

22. Geometry Modeling
Roof
Planar Structure

23. Model Refinement

24. Texture Mapping
Single image inevitably lack texture samples due to the foreshortening and occlusion.
But to achieve a good texture effect, there are 2 requirements:
1. the final texture should be consistent with the foreshortened image ;
2, the final texture should have consistent weathering pattern.
We need to know where is well textured and where not
Refine low quality region
Detect Texture Quality
Texture the occluded region

25. Texture Quality Detection
Back Project
Ratio = Triangle.size / imageProjection.size
Ratio > Threshold and Ratio is finite: large texture distortion
Ratio<Threshold: distortion free
Ratio is infinite: occluded
Texture in distortion free region will be used as the texture sample

26. Refinement for Low-Quality
Super- Resolution Problem

27. Occluded Region Texturing
The simplest way is to repeat the same texture as those of their symmetric counterparts, but this makes the model look artificial.
It is better to synthesize the texture in these region with common method
Another feature of the texture is weathering pattern, a constraint texture map is used according to the height of the architecture.
Input sample
Synthesized

28. Result

29. Result

30. Conclusion Contribution: Limitations: Novel Calibration Method
Texture Enhance Method
Limitations:
Strong assumption for simplification of camera calibration

31. Message from this Paper