Segmentation of Building Facades using Procedural Shape Priors Olivier Teboul, Loïc Simon Panagiotis Koutsourakis and Nikos Paragios

Introduction 3D Urban Modeling is of increasing interest Navigation systems, online applications, video games Huge amount of images available online (bing maps, google maps, flickr) Facade analysis plays a central role  semantic segmentation

Problem Rectified Facade image Segmentation into architectural classes

Problem Rectified Facade image Segmentation into architectural classes windows

Problem Rectified Facade image Segmentation into architectural classes windows walls

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky shops

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky shops Enforce architectural constraints

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky shops Enforce architectural constraints alignment

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky shops Enforce architectural constraints alignment consistent topology

Problem Rectified Facade image Segmentation into architectural classes windows walls balconies doors roofs sky shops Enforce architectural constraints alignment consistent topology procedural shape prior : The segmentation should be the result of a shape grammar derivation

Related Work Classic Segmentation methods Mean Shift, Level Set, MRF-based methods, Normalized Cut  Lack semantic information Semantic Segmentation methods He et al. CVPR 04, Shotton et al. ECCV 06  Do not guarantee architectural consistency ! Grammar-based methods Image-driven (bottom-up): Dick et al. ICCV 01, Müller et al. 07, Koutsourakis et al. 09  Based on self-similarity measure Grammar-driven (top-down): Alegre & Dellaert 04 , Ripperda et al. DAGM 06  Exploits the input grammar Recent Work: Vanegas et al. CVPR10, Toshev et al. CVPR10

Shape grammar [Stiny 72] Dictionary of shapes window wall balcony door roof sky shop facade floor groundfloor attic image tile

Shape grammar [Stiny 72] Dictionary of shapes Set of Replacement Rules window wall balcony door roof sky shop facade floor groundfloor attic image tile facade

Split Grammars [Wonka 03] A rule can only split a shape into several one along an axis Example: replace a floor by a sequence of walls and tiles A rule is characterized by the parameter vector W The dimension of W depends on the rule floor wall tile w1 w2 w3 w4 w5

Procedural Modeling of Facades Start from an axiom (Image) Sequentially apply replacement rules The derivation tree keeps track of the building structure

Procedural Modeling of Facades Start from an axiom (Image) Sequentially apply replacement rules The derivation tree keeps track of the building structure

Procedural Modeling of Facades Start from an axiom (Image) Sequentially apply replacement rules The derivation tree keeps track of the building structure

Procedural Modeling of Facades Start from an axiom (Image) Sequentially apply replacement rules The derivation tree keeps track of the building structure

Procedural Modeling of Facades Start from an axiom (Image) Sequentially apply replacement rules The derivation tree keeps track of the building structure To be optimized : rule selection and split parameters

Segmentation energy

Segmentation energy Single Pixel x Feature vector  

Segmentation energy Single Pixel x

Segmentation energy Single Pixel x Single Region R R

Segmentation energy Single Pixel x Single Region R Segmentation π

Segmentation energy  

Pixel-wise MAP classification Supervised Learning Goal : learning a discriminative model of p(c|fx) Feature vector fx is a patch around x Randomized Forest classifiers [Breiman ML Journal 01, Lepetit & Fua PAMI 06] input Pixel-wise MAP classification Window probability Wall probability (red= 0 blue = 1) … Ground truth (color = class)

Optimization : grammar factorization Independent sampling of rules may lead to inconsistent buildings Idea : tie together the derivation of some architectural classes Factorization Independent derivation

Optimization : grammar factorization Reduces the dimension of the space of shapes Factorization is a natural way to fight the curse of dimensionality Allows a fixed representation of facades (independently from the layout topology). A segmentation is described by a fixed sequence of rules : 1 rule to split the image into sub-regions  r1 1 rule to split a façade into floors  r2 1 rule to split a floor into walls and tiles  r3 1 rules to split a tile into window and balcony  r4 … Segmentation π = (r1, …, rM)

Optimization: algorithm Start from an initial seed π0 = (r10, r20, …, rM0) π0

Optimization: algorithm Start from an initial seed π0 = (r10, r20, …, rM0) Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi) π0

Optimization: algorithm Start from an initial seed π0 = (r10, r20, …, rM0) Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi) Keep the best segmentation as new seed v π1

Optimization: algorithm Start from an initial seed π0 = (r10, r20, …, rM0) Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi) Keep the best segmentation as new seed Iterate N times

Segmentation with Procedural Shape Priors Quantitative Results Pixel-wise MAP 20 training images 10 test images Segmentation with Procedural Shape Priors

Analysis The shape grammar introduces a context : each class has fewer challengers Disambiguation thanks to grammar consistency : take advantage of the repetitions over the facade

Qualitative Results Architecturally consistent Robust to illumination conditions, hard cast shadow, reflections

Conclusion Procedural Shape Prior for Semantic Segmentation Grammar Factorization Source code for Randomized Forest http://www.mas.ecp.fr/vision/Personnel/teboul/source_code.html Database of Parisian facades http://www.mas.ecp.fr/vision/Personnel/teboul/data.html

Future Work Exploring more efficient optimization techniques Optimize the rule selection Robust algorithms inspired from grammar parsing Coupling with multiple views towards finer modeling Image-based modeling Other grammars and architectures Thank you!

Questions ?