1. Oxford Brookes Seminar Thursday 3 rd September, 2009 University College London1 Representing Object-level Knowledge for Segmentation and Image Parsing: Epitome Priors and Bayesian supervised clustering approaches Jonathan Warrell J.Warrell@cs.ucl.ac.uk

2. Talk Outline –A Typology of Segmentation and Parsing Problems – Supervised Parsing using Epitome Priors Scene Parsing Face Parsing –Object-level knowledge transfer for Unsupervised Parsing Segmentation, Unsupervised Parsing and object-level knowledge Bayesian Supervised Clustering –Summary University College London2Oxford Brookes Seminar Thursday 3 rd September, 2009

3. A Typology of Segmentation and Parsing Problems: Over-segmentation Over-segmentation University College London3Oxford Brookes Seminar Thursday 3 rd September, 2009 Normalized Cut (NC) Ren and Malik [ICCV 2003] Minimum Spanning Tree (FH) Felzenszwalb & Huttenlocher [IJCV 2004] a

4. A Typology of Segmentation and Parsing Problems: Segmentation as Grouping Segmentation as Perceptual Grouping University College London4Oxford Brookes Seminar Thursday 3 rd September, 2009 a Sample images with 3 human segmentations Martin et al [ICCV 2001]

5. A Typology of Segmentation and Parsing Problems: Segmentation as Grouping University College London5Oxford Brookes Seminar Thursday 3 rd September, 2009 a Original Image, DDMCMC result, human segmentation Tu and Zhu [PAMI 2002] Original Image, Result Ren and Malik [ICCV 2003] Sampling from a Posterior over Segmentations

6. A Typology of Segmentation and Parsing Problems: Segmentation as Grouping University College London6Oxford Brookes Seminar Thursday 3 rd September, 2009 a Original Image, UMC map, Segmentation at different thresholds Arabelaez et al [CVPR 2009] Creating a Hierarchy of Regions

7. A Typology of Segmentation and Parsing Problems: Single Object Segmentation Supervised Object Segmentation University College London7Oxford Brookes Seminar Thursday 3 rd September, 2009 a Bounding Box + Segmentation results on ETHZ shape database Gu et al [CVPR 2009]

8. A Typology of Segmentation and Parsing Problems: Single Object Segmentation Supervised Scene Parsing University College London8Oxford Brookes Seminar Thursday 3 rd September, 2009 a TextonBoost (Shotton et al [IJCV 2009]) Multiscale CRF (He and Zemel [CVPR 2004])

9. A Typology of Segmentation and Parsing Problems: Object Discovery Co-Segmentation / Object Discovery University College London9Oxford Brookes Seminar Thursday 3 rd September, 2009 a Russell et al [CVPR 2006]

10. A Typology of Segmentation and Parsing Problems: General Unsupervised Parsing General Unsupervised Parsing University College London10Oxford Brookes Seminar Thursday 3 rd September, 2009 a Li et al [CVPR 2009]

11. A Typology of Segmentation and Parsing Problems: Summary Summary University College London11Oxford Brookes Seminar Thursday 3 rd September, 2009 a

12. A Typology of Segmentation and Parsing Problems: Summary Summary University College London12Oxford Brookes Seminar Thursday 3 rd September, 2009 a

13. A Typology of Segmentation and Parsing Problems: Summary Summary University College London13Oxford Brookes Seminar Thursday 3 rd September, 2009 a

14. Talk Outline University College London14Oxford Brookes Seminar Thursday 3 rd September, 2009 Talk Outline –A Typology of Segmentation and Parsing Problems – Supervised Parsing using Epitome Priors Scene Parsing Face Parsing –Object-level knowledge transfer for Unsupervised Parsing Segmentation, Unsupervised Parsing and object-level knowledge Bayesian Supervised Clustering –Summary

15. A Simple Scene Parsing Pipeline Supervised Parsing Using Epitome Priors: A Simple Scene Parsing Pipeline University College London15Oxford Brookes Seminar Thursday 3 rd September, 2009 Local Classifier Integration of prior/ contextual knowledge

16. A Simple Scene Parsing Pipeline Supervised Parsing Using Epitome Priors: A Simple Scene Parsing Pipeline University College London16Oxford Brookes Seminar Thursday 3 rd September, 2009 Local Classifier Integration of prior/ contextual knowledge Includes: Likely configurations of objects General properties of the label field e.g. stationarity/non-stationarity, symmetry Object shape Necessary for: Disambiguation Correction of classifier errors

17. Local MRF/CRF approaches Supervised Parsing Using Epitome Priors: Local MRF/CRF approaches University College London17Oxford Brookes Seminar Thursday 3 rd September, 2009 Geman and Geman, PAMI, 1986 Kumar and Herbert, ICCV, 2003 Shotton et al, ECCV, 2006

18. Local MRF/CRF approaches Supervised Parsing Using Epitome Priors: Local MRF/CRF approaches University College London18Oxford Brookes Seminar Thursday 3 rd September, 2009 Geman and Geman, PAMI, 1986 Kumar and Herbert, ICCV, 2003 Shotton et al, ECCV, 2006 + Few Parameters + Efficient Inference (via alpha expansion)

19. Local MRF/CRF approaches Supervised Parsing Using Epitome Priors: Local MRF/CRF approaches University College London19Oxford Brookes Seminar Thursday 3 rd September, 2009 Geman and Geman, PAMI, 1986 Kumar and Herbert, ICCV, 2003 Shotton et al, ECCV, 2006 + Few Parameters + Efficient Inference (via alpha expansion) – Limited representation ability (e.g. more than 2 objects, object shape)

20. Larger Clique Size CRF approaches Supervised Parsing Using Epitome Priors: Larger Clique Size CRF approaches University College London20Oxford Brookes Seminar Thursday 3 rd September, 2009 He et al, CVPR, 2004 Kohli et al, CVPR, 2007

21. Larger Clique Size CRF approaches Supervised Parsing Using Epitome Priors: Larger Clique Size CRF approaches University College London21Oxford Brookes Seminar Thursday 3 rd September, 2009 He et al, CVPR, 2004 Kohli et al, CVPR, 2007 + Rich representation – Sampling required during training and inference

22. Larger Clique Size CRF approaches Supervised Parsing Using Epitome Priors: Larger Clique Size CRF approaches University College London22Oxford Brookes Seminar Thursday 3 rd September, 2009 He et al, CVPR, 2004 Kohli et al, CVPR, 2007 + Efficient algorithms for inference (alpha expansion, s-t cut) – Constrained representation ability

23. Directed Models Supervised Parsing Using Epitome Priors: Directed Models University College London23Oxford Brookes Seminar Thursday 3 rd September, 2009 Domke et al, CVPR, 2008 Feng et al, PAMI, 2002

24. Directed Models Supervised Parsing Using Epitome Priors: Directed Models University College London24Oxford Brookes Seminar Thursday 3 rd September, 2009 Domke et al, CVPR, 2008 Feng et al, PAMI, 2002 + Efficient algorithms for training and inference +/– Mild impositions on representation – Large increase in parameters needed for a non-stationary distribution

25. Supervised Parsing Using Epitome Priors: Summary of problems University College London25Oxford Brookes Seminar Thursday 3 rd September, 2009 Summary of problems –Trade-off between 1) representational ability and 2) efficiency of training and inference algorithms –Desirability of modeling non-stationary distributions with limited parameters

26. Summary of problems –Trade-off between 1) representational ability and 2) efficiency of training and inference algorithms –Desirability of modeling non-stationary distributions with limited parameters Advantages of Epitomes –Parameterization is compact –Efficient training and inference –Non-stationary distributions easily modeled Supervised Parsing Using Epitome Priors: Summary of problems University College London26Oxford Brookes Seminar Thursday 3 rd September, 2009

27. Epitomic Analysis of Appearance and Shape Jojic, Frey and Kannan, 2003 Epitome as a model of Image Patches University College London27 Jojic, Frey and Kannan, Epitomic Analysis of Appearance and Shape [ICCV 2003] Epitome Parameters:{α, μ, σ} Oxford Brookes Seminar Thursday 3 rd September, 2009

28. Epitomes over Label Patches (Discrete) Epitomes over Label Patches University College London28 Epitome Parameters:{α, θ} Oxford Brookes Seminar Thursday 3 rd September, 2009

29. Generating from a Mixture of Multinomials University College London29 α: θ: l: h … Oxford Brookes Seminar Thursday 3 rd September, 2009

30. Generating from a Mixture of Multinomials University College London30 α: θ: l: h=2 … Oxford Brookes Seminar Thursday 3 rd September, 2009

31. Generating from a Mixture of Multinomials University College London31 α: θ: l: h=2 … Oxford Brookes Seminar Thursday 3 rd September, 2009

32. Generating from a Mixture of Multinomials University College London32 α: θ: l: h=1 … Oxford Brookes Seminar Thursday 3 rd September, 2009

33. Generating from a Mixture of Multinomials University College London33 α: θ: l: h=1 … Oxford Brookes Seminar Thursday 3 rd September, 2009

34. Generating from a Mixture of Multinomials University College London34 α: θ: l: h=1 … Oxford Brookes Seminar Thursday 3 rd September, 2009

35. Generating from a Mixture of Multinomials University College London35 α: θ: l: h=1 … Oxford Brookes Seminar Thursday 3 rd September, 2009

36. Generating from an Epitomized Mixture of Multinomials University College London36 l: h … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

37. Generating from an Epitomized Mixture of Multinomials University College London37 l: h=3 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

38. Generating from an Epitomized Mixture of Multinomials University College London38 l: h=3 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

39. Generating from an Epitomized Mixture of Multinomials University College London39 l: h=4 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

40. Generating from an Epitomized Mixture of Multinomials University College London40 l: h=4 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

41. Generating from an Epitomized Mixture of Multinomials University College London41 l: h=6 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

42. Generating from an Epitomized Mixture of Multinomials University College London42 l: h=6 … α: θ: Oxford Brookes Seminar Thursday 3 rd September, 2009

43. Epitomic Analysis of Appearance and Shape Jojic, Frey and Kannan, 2003 Epitome as model of a whole image University College London43 x:… … z: {α, μ, σ} See Jojic, Frey and Kannan, Epitomic Analysis of Appearance and Shape [ICCV 2003] for full model. Oxford Brookes Seminar Thursday 3 rd September, 2009

44. The Epitome Tree (an epitomized TSBN) The Epitome Tree (an epitomized Tree Structured Belief Network) University College London44 h1h1 h 2,1 h 2,2 h 3,1 h 3,2 h 3,3 h 3,4 l1l1 l2l2 … α θ1θ1 θ2θ2 θ3θ3 For Tree Structured Belief Networks, see Feng, Williams and Felderhof. Combining Belief Networks and Neural Networks for Scene Segmentation. PAMI, 2002 Oxford Brookes Seminar Thursday 3 rd September, 2009

45. CRF Model with Epitome Potentials for Inference University College London45 Models trained generatively, using E-M CRF model used for inference, with Epitome potentials Epitomized Mixture of Multinomials Epitome Tree Alternative training: Contrastive Divergence (see Hinton, Training Products of Experts by Contrastive Divergence, Neural Comp. 2002.) Oxford Brookes Seminar Thursday 3 rd September, 2009

46. Epitomized Priors: Results on Corel Epitomized Priors: Results on Corel (CVPR 2009) University College London46 ImageGround Truth TextonBoost Unary TextonBoost + epitome tree Oxford Brookes Seminar Thursday 3 rd September, 2009

47. Epitomized Priors: Results continued Results: comparing models on Corel and Sowerby datasets Comparison with TextonBoost on Corel University College London47 [1] Shotton, Winn, Rother and Criminisi. TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation. ECCV, 2006 Oxford Brookes Seminar Thursday 3 rd September, 2009

48. Supervised Parsing Using Epitome Priors: Face Parsing Face Parsing University College London48Oxford Brookes Seminar Thursday 3 rd September, 2009 Labeled Faces in the Wild (Huang et al, 2007)

49. Supervised Parsing Using Epitome Priors: Face Parsing Models University College London49Oxford Brookes Seminar Thursday 3 rd September, 2009 Mixture of Multinomials Epitome Model Epitome Model with Spatial Weighting Epitome Tree (samples from prior)

50. Supervised Parsing Using Epitome Priors: Face Parsing Results University College London50Oxford Brookes Seminar Thursday 3 rd September, 2009 Comparing: A) original image, B) unary classifier, C) weighted epitome, D) epitome tree, E) ground truth Face Parsing Results (ICIP 2009)

51. Supervised Parsing Using Epitome Priors: Face Parsing Results University College London51Oxford Brookes Seminar Thursday 3 rd September, 2009 Face Parsing Results Overall pixels correct: F-measure on individual classes: [1] Shotton, Winn, Rother and Criminisi. TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation. ECCV, 2006

52. Talk Outline University College London52Oxford Brookes Seminar Thursday 3 rd September, 2009 Talk Outline –A Typology of Segmentation and Parsing Problems – Supervised Parsing using Epitome Priors Scene Parsing Face Parsing –Object-level knowledge transfer for Unsupervised Parsing Segmentation, Unsupervised Parsing and object-level knowledge Bayesian Supervised Clustering –Summary

53. Object-level knowledge transfer for Unsupervised Parsing Object-level knowledge for Segmentation? University College London53Oxford Brookes Seminar Thursday 3 rd September, 2009

54. Object-level knowledge transfer for Unsupervised Parsing Object-level knowledge for Segmentation? University College London54Oxford Brookes Seminar Thursday 3 rd September, 2009

55. Object-level knowledge transfer for Unsupervised Parsing Object-level knowledge for Co-segmentation/Object Discovery? University College London55Oxford Brookes Seminar Thursday 3 rd September, 2009 ? ? ? (Indoors)

56. Object-level knowledge transfer for Unsupervised Parsing Object-level knowledge for Co-segmentation/Object Discovery? University College London56Oxford Brookes Seminar Thursday 3 rd September, 2009 ? ? ? (Jungle) (Indoors) (Country)

57. Object-level knowledge transfer for Unsupervised Parsing: Problem Summary University College London57Oxford Brookes Seminar Thursday 3 rd September, 2009 Problem Summary –Can we use abstract object knowledge to help with segmentation and unsupervised parsing tasks, where the test objects may be different to those seen in training? –Humans seem to be capable of this!

58. Object-level knowledge transfer for Unsupervised Parsing: Proposal University College London58Oxford Brookes Seminar Thursday 3 rd September, 2009 Proposal –Treat segmentation as a supervised clustering problem Ren and Malik [ICCV 2003] ‘Face Clustering’, using PLDA model see Prince et al [ICCV 2007] Bayesian Hierarchical Clustering, Heller and Ghahramani [ICML 2005]

59. Object-level knowledge transfer for Unsupervised Parsing: Bayesian Clustering Bayesian Supervised Clustering University College London59Oxford Brookes Seminar Thursday 3 rd September, 2009 x1:x1: y1:y1:y2:y2: x2:x2: xT:xT: yT:yT: … … … … … … [1 1 2 3 2 ][1 2 1 3 4 ] [ ? ] Model Learn: Pr(x|y), Pr(y) For a test point, seek: argmax Pr(y)Pr(x|y) y x1x1 x1x1 x1x1 123

60. Summary –Segmentation vs. Parsing: multiple ways of combining top-down and bottom-up information in a family of related tasks –Supervised Scene and Face parsing using Epitome Priors –Object-level knowledge transfer for Unsupervised Parsing (Bayesian supervised Clustering) University College London60Oxford Brookes Seminar Thursday 3 rd September, 2009

61. Any Questions? Details of the papers can be found at http://web4.cs.ucl.ac.uk/research/vis/pvl http://web4.cs.ucl.ac.uk/research/vis/pvl University College London61Oxford Brookes Seminar Thursday 3 rd September, 2009

62. Object-level knowledge transfer for Unsupervised Parsing: Mixture of Gaussians Example Mixture of Gaussians Example –Introduce a hidden variable, h n, associated with each observation, taking values 1…C (N<C) –Let: P(x n |h n = c) = G(x|μ c,Σ c ) –Generative process now becomes: Sample y: [1 1 2 3 3 2 2 1 3 … ] (from Pr(y)) (max value, K) Sample h: [4 4 5 2 2 5 5 4 2 … ] (from Pr(h)) Sample x: [x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9... ] (from Pr(x|h)) –Likelihood model is: University College London62Oxford Brookes Seminar Thursday 3 rd September, 2009 (function ind(i,j) returns the vector index of the j th entry of y with value i)

63. Object-level knowledge transfer for Unsupervised Parsing: Factor Analysis and PLDA examples Factor Analysis Extension –Same as Mixture of Gaussians, but now with a continuous h: Probabilistic Linear Discriminant Analysis –Introduce an additional within-cluster hidden variable, w: University College London63Oxford Brookes Seminar Thursday 3 rd September, 2009

64. Object-level knowledge transfer for Unsupervised Parsing: Prior and Inference The prior on y –Prefer a certain number of clusters? –Smoothness constraints? / Tree-based Region Prior? –Image consistency constraints? University College London64Oxford Brookes Seminar Thursday 3 rd September, 2009 Inference –Pairwise merging (within/between images) –Tree-based merging –Random subset selection –MCMC search