Procedural Modeling of Architectures towards 3D Reconstruction Nikos Paragios Ecole Centrale Paris / INRIA Saclay Ile-de-France Joint Work: P. Koutsourakis, L. Simon, O. Teboul & N. Komodakis

Outline The language of Architecture Automatic generation of buildings Toward single image-based reconstruction Discrete MRFs, LP and Duality 04/11/0810/04/08

Part I: Shape Grammars

3D reconstruction Emerging technology in information society – Post-production, games, navigation Current Methods: image based – Not scalable, computational expensive, low level, 04/11/0810/04/08

The Geometry of Architecture Very complex geometry… …but deeply structured due to: – Laws – Practical constraints – Economical constraints – Aesthetic motivation Each architectural style is constrained differently. But buildings of the same style obey the same rules. 04/11/0810/04/08

Simple example of structure 04/11/0810/04/08

Architecture as a large scale Lego Architecture is made of repetitions of basic elements, place holders or real elements. – Facades, floors, windows, balconies, doors… A specific style can be describe by : – The specification of the repetitions (geometry)‏ – The nature of the elements (semantics)‏ 04/11/0810/04/08

Shape Grammars Procedural modeling is a set of techniques for creating 3D models from a set of rules. Traditional 3D modeling can be very time consuming for large models. Large scale architectural models are especially suited for this kind of techniques. There is a strong analogy with classical string grammars. 04/11/0810/04/08

Basic Shapes and Scopes The Basic Shapes are for Shape Grammars what Symbols are for String Grammars A Basic Shape is composed of – A Semantic – A fixed geometry – Appearance attributes A Scope provides the orientation, scaling, position of a basic shape 04/11/0810/04/08

Shape 04/11/0810/04/08 The shape is a hierarchical representation (a tree) of the building. Each node holds a basic shape with an associated scope.

Rules : Interacting with shapes A rule can be described as: precondition : LHS  RHS Given a precondition on the context, a LHS shape is replaced by the RHS shape. From the hierarchical point of view some children are added to the node LHS, increasing the depth of the tree. Rules are built from operators 04/11/0810/04/08

The operators of the grammar Except the roof operators (which rely on computational geometry) the operators are fairly simple. – Transformation : rotation,translation, scaling – Split – Repeat – Mirror – Component Split – Roof operators : hipped, mansard 04/11/0810/04/08

Operators continued Splits subdivide a scope along an axis Component split decomposes a mesh into faces Roof operators are based on Weighted Straight Skeleton algorithm 04/11/0810/04/08

Step by step generation 04/11/0810/04/08

Generating : from details to large scale 04/11/0810/04/08

Demos Video\ComplexBuilding.ogg video\roofsAndTexture.ogg 04/11/0810/04/08

Grammar-based reconstruction A building is not described by a geometrical attributes anymore Building = sequence of rules Goal : optimize the sequence of rules that best explains a given facade image 04/11/0810/04/08

Model Quality Evaluation Assuming that the camera has been weakly calibrated, the elements of the resulting model can be reprojected on the image. The score of the model can be defined in terms of a distance between the observed regions and the known semantics. 04/11/0810/04/08

Part II: MRF optimization via the primal-dual schema

The MRF optimization problem vertices G = set of objects edges E = object relationships set L = discrete set of labels V p (x p ) = cost of assigning label x p to vertex p (also called single node potential) ‏ V pq (x p,x q ) = cost of assigning labels (x p,x q ) to neighboring vertices (p,q) (also called pairwise potential) ‏ Find labels that minimize the MRF energy (i.e., the sum of all potentials):

MRF hardness MRF pairwise potential MRF hardness linear exact global optimum arbitrary local optimum metric global optimum approximation Move right in the horizontal axis, But we want to be able to do that efficiently, i.e. fast and remain low in the vertical axis (i.e., still be able to provide approximately optimal solutions) ‏

The primal-dual schema  Say we seek an optimal solution x* to the following integer program (this is our primal problem): (NP-hard problem) ‏  To find an approximate solution, we first relax the integrality constraints to get a primal & a dual linear program: primal LP: dual LP:

The primal-dual schema Goal: find integral-primal solution x, feasible dual solution y such that their primal-dual costs are “close enough”, e.g., primal cost of solution x primal cost of solution x dual cost of solution y dual cost of solution y cost of optimal integral solution x* cost of optimal integral solution x* Then x is an f * -approximation to optimal solution x*

The primal-dual schema sequence of dual costs sequence of primal costs … unknown optimum … The primal-dual schema works iteratively Global effects, through local improvements! Instead of working directly with costs (usually not easy), use RELAXED complementary slackness conditions (easier) ‏ Different relaxations of complementary slackness Different approximation algorithms!!!

(only one label assigned per vertex) enforce consistency between variables x p,a, x q,b and variable x pq,ab The primal-dual schema for MRFs Binary variables x p,a =1 label a is assigned to node p x pq,ab =1 labels a, b are assigned to nodes p, q x p,a =1 label a is assigned to node p x pq,ab =1 labels a, b are assigned to nodes p, q

The primal-dual schema for MRFs During the PD schema for MRFs, it turns out that: each update of primal and dual variables solving max-flow in appropriately constructed graph Max-flow graph defined from current primal-dual pair (x k,y k )  (x k,y k ) defines connectivity of max-flow graph  (x k,y k ) defines capacities of max-flow graph Max-flow graph is thus continuously updated Resulting flows tell us how to update both:  the dual variables, as well as  the primal variables for each iteration of primal-dual schema

primal-dual framework Handles wide class of MRFs Approximately optimal solutions Theoretical guarantees AND tight certificates per instance Significant speed-up for static MRFs Significant speed-up for dynamic MRFs - New theorems - New insights into existing techniques - New view on MRFs

04/11/0810/04/08

Future Work Efficient optimization framework, through neuro- dynamic programming like methods Image-based grammar learning Augmentation of the expressiveness of the grammar Large scale representation of forests of buildings 04/11/0810/04/08

Questions ? 04/11/0810/04/08