1. Texture We would like to thank Amnon Drory for this deck הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.

2. Syllabus Textons TextonsBoost

3. Textons Run filter bank on images Build Texton dictionary using K-means Map texture image to histogram Histogram Similarity using Chi-square

4. TextonBoost Build Texton dictionary Texture Layout (pixel, rectangle, Texton) Count number of textons in rectangle Use Integral Image Generate multiple Texture layouts (Features) For each class do 1-vs-all classifier: – For each pixel in class Train GentleBoost Classifier Map strong classifier to probability Take Maximum value

5. CRF/MRF How to ensure Spatial Consistency? Bayes LikelihoodPosterior ML MAP Prior

6. Semantic Texton Forest Decision Trees Forest and Averaging Split decision to minimize Entropy Two level STF to add spatial regularization Works well when there is ample data, does not generalize well

7. (1) Textons B. Julesz, Leung, Malik M. Varma, A. Zisserman (II) TextonBoost J. Shotton, J. Winn, C. Rother, A. Criminisi (III) Semantic Texton Forests J. Shotton, M. Johnson, R. Cipolla (IV) Pose Recognition from Depth Images J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake

8. Textures

9. Filter Bank

10. K-means

11. Texton Histogram

12. Classification

13. Results

14. TextonBoost : Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation J. Shotton *, J. Winn †, C. Rother †, and A. Criminisi † * University of Cambridge † Microsoft Research Ltd, Cambridge, UK

15. TextonBoost Simultaneous recognition and segmentation Simultaneous recognition and segmentation Explain every pixel Explain every pixel

16. TextonBoost Input: Input: 1. Training: Images with pixel level ground truth classification MSRC 21 Database

17. TextonBoost Input: Input: 1. Training: Images with pixel level ground truth classification. 2. Testing: Images Output: Output: A classification of each pixel in the test images to an object class.

18. Conditional Random Field Unary TermBinary Term Unary TermBinary Term

19. Textons Shape filters use texton maps ( [Varma & Zisserman IJCV 05], [Leung & Malik IJCV 01]) Convolve with 17D filter bank (Gaussians, Derivatives of Gaussians, DoGs, LoGs) – Can use Gabor instead Use k-means to create 400 clusters Compact and efficient characterisation of local texture Texton map Colors  Texton Indices Input image  Clustering Filter Bank

20. CRF: Unary Term 0.10.020.230.470.001 Probability of class c_i given feature vector x

21. Texture-Layout Filters Pair: Feature responses v(i, r, t) Large bounding boxes enable long range interactions rectangle rtexton t (, ) v(i 1, r, t) = a v(i 3, r, t) = a/2 up to 200 pixels (, ) (, )

22. Texture Layout (Toy Example)

23. CRF: Binary Term

24. Potts model Potts model encourages neighbouring pixels to have same label encourages neighbouring pixels to have same label Contrast sensitivity Contrast sensitivity encourages segmentation to follow image edges encourages segmentation to follow image edges

25. Accurate Segmentation? Boosted classifier alone – effectively recognises objects – but not sufficient for pixel- perfect segmentation Conditional Random Field (CRF) – jointly classifies all pixels whilst respecting image edges unary term only CRF

26. The TextonBoost CRF Texture-LayoutColorlocation edge Unary Term Binary Term

27. Location Term Capture prior on absolute image location Capture prior on absolute image location treeskyroad Texture-LayoutColorlocation edge

28. Color Term Texture-LayoutColorlocation edge

29. Texture-Layout Term

30. Texton Boost - Summary Performs per-pixel classification using: 1. Statistics learned from Training Set: - Absolute location statistics - Configuration of textured areas around pixel of interest. 2. Cues from the Test Image: - Edges - Object Colors 3. Priors.

31. Results on 21-Class Database building

32. Effect of Model Components Shape-texture potentials only:69.6% + edge potentials:70.3% + Color potentials:72.0% + location potentials:72.2% shape-texture + edge + Color & location pixel-wise segmentation accuracies