1. Segmentation of Building Facades using Procedural Shape Priors
Olivier Teboul, Loïc Simon
Panagiotis Koutsourakis and Nikos Paragios

2. Introduction 3D Urban Modeling is of increasing interest
Navigation systems, online applications, video games
Huge amount of images available online (bing maps, google maps, flickr)
Facade analysis plays a central role  semantic segmentation

3. Problem
Rectified Facade image
Segmentation into architectural classes

4. Problem Rectified Facade image Segmentation into architectural classes
windows

5. Problem Rectified Facade image Segmentation into architectural classes
windows
walls

6. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies

7. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors

8. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs

9. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky

10. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky
shops

11. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky
shops
Enforce architectural constraints

12. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky
shops
Enforce architectural constraints
alignment

13. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky
shops
Enforce architectural constraints
alignment
consistent topology

14. Problem Rectified Facade image Segmentation into architectural classes
windows
walls
balconies
doors
roofs
sky
shops
Enforce architectural constraints
alignment
consistent topology
procedural shape prior :
The segmentation should be the result of a shape grammar derivation

15. Related Work Classic Segmentation methods
Mean Shift, Level Set, MRF-based methods, Normalized Cut
 Lack semantic information
Semantic Segmentation methods
He et al. CVPR 04, Shotton et al. ECCV  Do not guarantee architectural consistency !
Grammar-based methods
Image-driven (bottom-up): Dick et al. ICCV 01, Müller et al. 07, Koutsourakis et al. 09
 Based on self-similarity measure
Grammar-driven (top-down): Alegre & Dellaert 04 , Ripperda et al. DAGM 06
 Exploits the input grammar
Recent Work: Vanegas et al. CVPR10, Toshev et al. CVPR10

16. Shape grammar [Stiny 72] Dictionary of shapes window wall balcony door
roof
sky
shop
facade
floor
groundfloor
attic
image
tile

17. Shape grammar [Stiny 72] Dictionary of shapes Set of Replacement Rules
window
wall
balcony
door
roof
sky
shop
facade
floor
groundfloor
attic
image
tile
facade

18. Split Grammars [Wonka 03]
A rule can only split a shape into several one along an axis
Example: replace a floor by a sequence of walls and tiles
A rule is characterized by the parameter vector W
The dimension of W depends on the rule
floor
wall
tile
w1 w w w w5

19. Procedural Modeling of Facades
Start from an axiom (Image)
Sequentially apply replacement rules
The derivation tree keeps track of the building structure

20. Procedural Modeling of Facades
Start from an axiom (Image)
Sequentially apply replacement rules
The derivation tree keeps track of the building structure

21. Procedural Modeling of Facades
Start from an axiom (Image)
Sequentially apply replacement rules
The derivation tree keeps track of the building structure

22. Procedural Modeling of Facades
Start from an axiom (Image)
Sequentially apply replacement rules
The derivation tree keeps track of the building structure

23. Procedural Modeling of Facades
Start from an axiom (Image)
Sequentially apply replacement rules
The derivation tree keeps track of the building structure
To be optimized : rule selection and split parameters

24. Segmentation energy

25. Segmentation energy
Single Pixel x
Feature vector

26. Segmentation energy
Single Pixel x

27. Segmentation energy
Single Pixel x
Single Region R R

28. Segmentation energy
Single Pixel x
Single Region R
Segmentation π

29. Segmentation energy

30. Pixel-wise MAP classification
Supervised Learning
Goal : learning a discriminative model of p(c|fx)
Feature vector fx is a patch around x
Randomized Forest classifiers [Breiman ML Journal 01, Lepetit & Fua PAMI 06]
input
Pixel-wise MAP classification
Window probability
Wall probability
(red= 0 blue = 1) …
Ground truth
(color = class)

31. Optimization : grammar factorization
Independent sampling of rules may lead to inconsistent buildings
Idea : tie together the derivation of some architectural classes
Factorization
Independent derivation

32. Optimization : grammar factorization
Reduces the dimension of the space of shapes
Factorization is a natural way to fight the curse of dimensionality
Allows a fixed representation of facades (independently from the layout topology). A segmentation is described by a fixed sequence of rules :
1 rule to split the image into sub-regions  r1
1 rule to split a façade into floors  r2
1 rule to split a floor into walls and tiles  r3
1 rules to split a tile into window and balcony  r4 …
Segmentation π = (r1, …, rM)

33. Optimization: algorithm
Start from an initial seed π0 = (r10, r20, …, rM0) π0

34. Optimization: algorithm
Start from an initial seed π0 = (r10, r20, …, rM0)
Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi) π0

35. Optimization: algorithm
Start from an initial seed π0 = (r10, r20, …, rM0)
Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi)
Keep the best segmentation as new seed v π1

36. Optimization: algorithm
Start from an initial seed π0 = (r10, r20, …, rM0)
Gaussian perturbations of all the rules πi = (r1i, r2i, …, rMi)
Keep the best segmentation as new seed
Iterate N times

37. Segmentation with Procedural Shape Priors
Quantitative Results
Pixel-wise MAP
20 training images
10 test images
Segmentation with Procedural Shape Priors

38. Analysis
The shape grammar introduces a context : each class has fewer challengers
Disambiguation thanks to grammar consistency : take advantage of the repetitions over the facade

39. Qualitative Results Architecturally consistent
Robust to illumination conditions, hard cast shadow, reflections

40. Conclusion Procedural Shape Prior for Semantic Segmentation
Grammar Factorization
Source code for Randomized Forest
Database of Parisian facades

41. Future Work
Exploring more efficient optimization techniques
Optimize the rule selection
Robust algorithms inspired from grammar parsing
Coupling with multiple views towards finer modeling
Image-based modeling
Other grammars and architectures
Thank you!

42. Questions ?