1. M ACHINE L EARNING : F OUNDATIONS C OURSE TAU – 2012A P ROF. Y ISHAY M ANSOUR o TextonBoost : Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation J. Shotton*, J. Winn†, C. Rother†, and A. Criminisi† o * University of Cambridge o † Microsoft Research Ltd, Cambridge, UK Yaniv Bar March 2013

2. G OAL o Simultaneous recognition and segmentation: Efficiently detect a large number of object classes and give a pixel-perfect segmentation of an image into these classes.

4. D ATA AND C LASSES Original Paper: 3 DBs. Main DB: MSRC 21. MSRC 21-Class Object Recognition Database 591 hand-labelled images Original main DB was updated to MSRC 23. MSRC 23-Class Object Recognition Database 592 hand-labelled images

5. H IGH L EVEL A PPROACH High-level description of approach: Learn classifier based on relative texture locations for each class. Classification is then refined. Given an image, for each pixel: - Texture-Layout features are calculated - A boosting classifier gives the probability of the pixel belonging to each class - The discriminative model combines the boosting output with low-level color, location, and edge information; image receives final label.

6. Most important part of the model is the Shape/Context Potential – it is significant for object recognition and very rough segmentation results. Other potential such as Edge and Color refine the segmentation results. (a) Original image, (b) Shape, (c) (b)+edge, (d) (c)+color T EXTURE LAYOUT F EATURES

7. For modeling object shape, appearance and context we use a New texton-based features. This feature (texton) compact and efficient characterisation of local texture.

8. o The task is to recognize surfaces made from different materials on the basis of their texture appearance. o Different materials show different texture appearance. Moreover, texture appearance of the same material changes dramatically due to different viewpoint/lighting settings (specularities, shadows, and occlusions). W HAT A RE T EXTONS

9. Computing texton maps: Texton map Colours  Texton Indices Input image  Clustering Filter Bank Convolve 17-D filter bank (composed of gaussians, dogs, logs) with all training images Responses are clustered with K-means Each pixel is assigned a texton number C ALCULATING T EXTURE -L AYOUT FEATURES

10. Capturing appearance:

11. How Texture-Layout features jointly model texture and layout:

12. L EARNING Learning is done with Joint Boost algorithm – A version of Multi class gentle boost algorithm. I’ve used both AdaBoost.M1 and AdaBoost.Mh (multiclass reduction to binary which is due to the fact that AdaBoosting is only for binary classification).

13. T HE G OOD AND B AD The Good: Provides reasonable recognition + segmentation for many classes. Also, combines several good ideas. Most of previous works didn’t tackle the problem as a whole – rather, problems were treated separately. The Bad: Does not beat past work (in terms of quantitative recognition results) and a bit hacky.

14. C ODE -S EQUENCE OF EXECUTION 1. imagesTextonization.m (extract efficient images characterization) 2. calcModelFeatures.m (calculate the appearance (shape) potential context) 3. trainModel.m (build a classification model) 4. testModel.m (test the classification model with test data)

15. R ESULTS