Automatic 3D modelling of Architecture Anthony Dick 1 Phil Torr 2 Roberto Cipolla 1 1 Department of Engineering 2 Microsoft Research, University of Cambridge Cambridge

Automatic 3D modelling of architecture - BMVC'00 2 The goal Generate 3D models of architectural scenes from several images automatically –Including accurate geometry, texture Interactively built using Photobuilder! Available at +

Automatic 3D modelling of architecture - BMVC'00 3 Our approach Previous structure from motion algorithms use only image data We integrate image data with prior knowledge of architecture –The scene will be piecewise planar –Walls are likely to intersect at right angles –Walls are likely to be perpendicular to a common ground plane –Walls are likely contain doors and windows which have a highly constrained shape

Automatic 3D modelling of architecture - BMVC'00 4 Model representation Scene is modelled as a collection of “wall” planes Each wall plane has a plane equation and a boundary Each wall plane may contain offset layers such as doors, windows b c Front view a ar d (x,y) Overhead view Each offset layer is one of a collection of parameterised shapes

Automatic 3D modelling of architecture - BMVC'00 5 Model estimation Structure estimation has 2 parts: –How many walls are in the scene and what are their parameters? –How many offset layers does each wall contain, what shape are they, and what are their parameters? [ECCV2000] Model selection between different shapes

Automatic 3D modelling of architecture - BMVC'00 6 Previous work Manually defined homography Initialise offset layer estimates using dense correspondence Fit 4 different shape models to each region –Use Bayesian model selection criterion to select best shape model Initial After model fitting + selection

Automatic 3D modelling of architecture - BMVC'00 7 What’s new Extension to scenes with multiple wall planes Automatic segmentation of walls

Automatic 3D modelling of architecture - BMVC'00 8 Initialisation Feature-based structure from motion –Track points –Estimate pairwise epipolar geometry –Camera self-calibration [Mendonca CVPR99]

Automatic 3D modelling of architecture - BMVC'00 9 Plane segmentation Recursive RANSAC plane extraction Assume all planes perpendicular to common ground plane Project onto ground plane Derive plane boundaries perpendicular and parallel to ground plane Reconstruction projected onto ground plane

Automatic 3D modelling of architecture - BMVC'00 10 Optimising the planar model Gradient descent search –Cost function: SSE of model projected into each image Parameters to vary: –Ground plane orientation –Boundary and intersection points of each plane Before fitting After fitting

Automatic 3D modelling of architecture - BMVC'00 11 Evaluating the cost function Search requires many evaluations of cost function This is expensive Green’s Theorem: Sum vector field A around region boundary Cache results for best efficiency R1 R2 Cost of R2, L(R2) = L(R1) – L(e1) – L(e2) + L(e3) + L(e4) e1 e2 e3 e4 e1 e2

Automatic 3D modelling of architecture - BMVC'00 12 Results Courtyard corner

Automatic 3D modelling of architecture - BMVC'00 13 The “castle” sequence Images from

Automatic 3D modelling of architecture - BMVC'00 14 Future work Use of lines to initialise offset layers –Join nearby lines into rectangles –Use knowledge of window height/width ratios More extensive and structured set of shapes –Rather than simply testing each possibility –Possible use of architectural shape grammars And in conclusion… –General framework of combining prior knowledge and image data is a useful one –Challenge is to formulate prior knowledge usefully

Automatic 3D modelling of architecture - BMVC'00 15 The Bayesian framework constan t model priorevidence likelihood prior Bayes Rule: Evidence: Model parameters  Wall planes: plane boundary, plane equations Offset layers: Height, width, x, y position

Automatic 3D modelling of architecture - BMVC'00 16 Having optimised for main walls, want to fit doors, windows etc. This is the same problem tackled earlier, but initialisation is now more difficult –Each plane covers less of the image –There may be some fitting errors Manually set number of primitives on each plane –Assumes evenly spaced, vertically centred –Fits each model from this initialisation Planar parallax