Unsupervised Learning of Hierarchical Spatial Structures Devi Parikh, Larry Zitnick and Tsuhan Chen

2 … hierarchical spatial patterns Our visual world… What is an object? What is context? Intro Approach Results Conclusion

3 Goal Unsupervised! Intro Approach Results Conclusion

4 Related work [Todorovic 2008] [Fidler 2007] [Zhu 2008] [Sivic 2008] Fully unsupervised Structure and parameters learnt From features to multiple objects Intro Approach Results Conclusion

5 Model Rule based c2c2 c4c4 c1c1 c2c2 c3c3 r1r Intro Approach Results Conclusion

6 c2c2 r2r2 c1c1 c2c2 c3c3 r1r Model Rule based Intro Approach Results Conclusion

7 c2c2 r2r2 c1c1 c2c2 c3c3 r1r Model Hierarchical rule-based Intro Approach Results Conclusion

8  Rules R  Image-parts V Model  Codewords C  Features F Intro Approach Results Conclusion

9 Model  Notation V = {v} instantiated image-parts r v rule corresponding to instantiated part v Ch(r v ) = {x} children of rule r v  includes instantiated children Ch(v)  and un-instantiated children Intro Approach Results Conclusion

10 Model Intro Approach Results Conclusion

11 Inference Intro Approach Results Conclusion

12 Inference Intro Approach Results Conclusion

13 Inference Intro Approach Results Conclusion

14 Inference Intro Approach Results Conclusion

15 Inference Intro Approach Results Conclusion

16 Inference Intro Approach Results Conclusion

17 Inference Intro Approach Results Conclusion

18 Inference Intro Approach Results Conclusion

19 Inference Intro Approach Results Conclusion

20 Inference Intro Approach Results Conclusion

21 Minimum Cost Steiner Tree Charikar 1998 Inference Intro Approach Results Conclusion

22 Inference Intro Approach Results Conclusion

23 Generalized distance transform Felzenszwalb et al Inference Intro Approach Results Conclusion

24  EM style  Initialize rules  Infer rules  Update parameters  Modify rules Learning Intro Approach Results Conclusion

25  Initialize rules … Learning Intro Approach Results Conclusion

26  Inference … Learning Intro Approach Results Conclusion

27  Inference … Learning Intro Approach Results Conclusion

28  Add children … Learning Intro Approach Results Conclusion

29  Add children  Update parameters  Pruning children  Removing rules … Learning Intro Approach Results Conclusion

30  Adding rules Randomly add rules … … Learning Intro Approach Results Conclusion

31 Behavior  Competition among rules  Competition with root (noise) Intro Approach Results Conclusion

32 Behavior  Competition among rules  Competition with root (noise)  Dropping children and rules  Number of children  Structure of DAG and tree  # rules, parameters, structure learnt automatically  Multiple instantiations of rules  Multiple children with same appearance Intro Approach Results Conclusion

Experiment 1: Faces & Motorbikes Intro Approach Results Conclusion

34  Faces and Motorbikes  SIFT (200 words)  Learnt 15 L1 rules, 2 L2 rules  Each L1 rule  average ~7 children  Each L2 rule  average ~4 children Faces & Motorbikes Intro Approach Results Conclusion

35 Example rules Intro Approach Results Conclusion

36 Patches Intro Approach Results Conclusion

37 Localization behavior Intro Approach Results Conclusion

38 Categorization behavior Faces Motorbikes Faces Motorbikes Faces Motorbikes occurrence code-words first level rules second level rules Intro Approach Results Conclusion

39 Categorization behavior Words RulesTree Words: 94 % Tree: 100% Kmeans PLSA SVM Intro Approach Results Conclusion

40 Edge features Words: 55 % Tree: 82% Intro Approach Results Conclusion

Experiment 2: Six categories Intro Approach Results Conclusion

42 Six categories 61 L1 rules (~9 children) 12 L2 rules (~3 children) Kim 2008: 95 % Words: 87 % Tree: 95 % Intro Approach Results Conclusion

Experiment 3: Scene categories Intro Approach Results Conclusion

44 Scene categories Image Segmentation Mean color Codeword Intro Approach Results Conclusion

45 Outdoor scenes rules images Intro Approach Results Conclusion

Experiment 4: Structured street scenes Intro Approach Results Conclusion

47 Windows Intro Approach Results Conclusion

48 Object categories Intro Approach Results Conclusion

49 Object categories Intro Approach Results Conclusion

50 Object categories Intro Approach Results Conclusion

51 Parts of objects Intro Approach Results Conclusion

52 Multiple objects Intro Approach Results Conclusion

53 Street Scenes (PLSA) Intro Approach Results Conclusion

54 Dataset specific rules irrelevant relevant Intro Approach Results Conclusion

55 Conclusion  Unsupervised learning of hierarchical spatial patterns  Low level features, object parts, objects, regions in scene  Rule-based approach  Learning: EM style  Inference: Minimum cost Steiner tree  Features  SIFT, edges, color segments Intro Approach Results Conclusion

56 Summary I Root Scene Objects Object Parts Features Intro Approach Results Conclusion