Towards Total Scene Understanding: Classification, Annotation and Segmentation in an Automatic Framework Li-Jia Li, Richard Socher, Li Fei- Fei 1

2 City Travel Pagoda Sunrise Sunshine Sun

3 City Travel Pagoda Sunrise Sunshine Sun Weber et al 00 Fergus et al 03 Felzenswalb et al 04 Fei-Fei et al 05 Sivic et al 05 Bosch et al 06 Oliva et al 01 Lazebnik et al 06 Shi et al 00 Felzenszwalb et al04 Sali et al 99 Winn et al 05 Kumar et al 05 Cao et al 07 Russell et al 06 Todorovic et al 06 Duygulu et al 02 Barnard et al 03 Blei et al 03 Gupta et al 08 Alipr Li et al 03 Sudderth et al 05 Segmentation Classification Annotation Remark: Approaches in yellow will be used to compare with our model in later Experiments.

4 City Travel Pagoda Sunrise Sunshine Sun Weber et al 00 Fergus et al 03 Felzenswalb et al 04 Fei-Fei et al 05 Sivic et al 05 Bosch et al 06 Oliva et al 01 Lazebnik et al 06 Shi et al 00 Felzenszwalb et al04 Sali et al 99 Winn et al 05 Kumar et al 05 Cao et al 07 Russell et al 06 Todorovic et al 06 Duygulu et al 02 Barnard et al 03 Blei et al 03 Gupta et al 08 Alipr Li et al 03 Sudderth et al 05 Segmentation Classification Annotation Total Scene Understanding

Application 5

6 ClassificationAnnotationSegmentation Mutually beneficial!

7 Athlete Horse Grass Trees Sky Saddle ClassificationAnnotationSegmentation Horse class: Polo

8 Horse Sky Tree Grass Athlete Horse Grass Trees Sky Saddle ClassificationAnnotationSegmentation Horse Athlete class: Polo

9 Horse Athlete Horse Grass Trees Sky Saddle ClassificationAnnotationSegmentation

10 Related Work: Tu et al 03 Annotation Segmentation Horse Sky Tree Grass Horse Athlete Li & Fei-Fei 07 Annotation Classification Sky Grass Horse Athlete Horse Class: Polo Classification Segmentation Tree Heitz et al 08 Class: Polo

Learning Model Recognition & Experiment Outline Classification Annotation Segmentation

12 C Nr O R NFNF X ArAr Nt Z S T D Athlete Horse Grass Trees Sky Saddle

13 C Visual Text class: Polo Athlete Horse Grass Trees Sky Saddle Joint distribution of random variable Visual Component Text Component. D

14 O Text Component. D Visual Text C class: Polo

15 R NFNF Color Location Texture Shape Text Component. O D Visual Text C class: Polo

R NFNF O D Visual Text C class: Polo 16 X ArAr Text Component.

R NFNF O D Visual Text C class: Polo X ArAr Z NrNt “Connector variable” Athlete Horse Grass Trees Sky Saddle Text Component.

R NFNF O D Visual Text C class: Polo X ArAr Z NrNt “Connector variable”. S Athlete Horse Grass Trees Sky Saddle Athlete Horse Grass Trees Sky Saddle Visible Not visible “Switch variable” Horse Athlete Horse

R NFNF O D Visual Text C class: Polo X ArAr Z NrNt “Connector variable” S Athlete Horse Grass Trees Sky Saddle Visible Not visible “Switch variable” T Horse.

Visual Text C Nr O R NF X Ar Nt Z S T Learning Model Recognition & Experiment Outline

21 Learning Exact Inference is Intractable ! Relationship of the random variables Visual Text C Nr O R NF X Ar Nt Z S T

22 Relationship of the random variables Visual Text C Nr O R NF X Ar Nt Z S T Top-down force Bottom-up force from visual information Bottom-up force from text information Collapsed Gibbs Sampling (R. Neal, 2000)

Scene/Event images from the Internet There is no object-text correspondence… Athlete Horse Grass Tree Saddle 23

Scene/Event images from the Internet Our model builds the correspondence… C Nr O R NFNF X ArAr Nt Z S T D Athlete Horse Grass Tree Saddle 24

25 Athlete Horse Grass Trees Sky Saddle Athlete Horse Grass Ball However, a big obstacle is: many objects always co-occur together ? ? ? Scene/Event images from the Internet

26 C R NFNF X Ar Nr Z Nt T S O One solution: some good initialization of O Grass Athlete Horse Athlete Horse Grass Trees Sky Saddle Scene/Event images from the Internet

Scene/Event images from the Internet 27 Initializing O: obtain internet images for each O Object images

28 Scene/Event images C R NFNF X Ar Nr Z Nt T S O Any object detection & segmentation Algorithm D Initializing O: train an object detector for each O Object images Event/Scene images

29 Scene/Event images … Black box object detection & segmentation Black box object detection & segmentation C R NFNF X Ar Nr Z Nt T S O D Initialize O in the scene image by the trained object detectors Object images Event/Scene images Any object detection & segmentation Algorithm

30 Scene/Event images … Black box object detection & segmentation Black box object detection & segmentation C R NFNF X Ar Nr Z Nt T S O Black box object detection & segmentation D Initialize O in the scene image by the trained object detectors Cao & Fei-Fei, 2007 θ C X R O Nr Ar Our Model Object images Event/Scene images

C R NFNF X Ar Nr Z Nt T S O D Auto - Auto -semi-supervised learning: Small # of initialized images + Large # of uninitialized images Our Model + Athlete Horse Grass Tree Saddle Wind Small # of initialized images Athlete Rock Grass Tree Sky Rope Athlete Snow Tree Sky Snowboard Large # of uninitialized images Scene/Event images

Athlete Horse Grass Tree Saddle Wind Athlete Rock Grass Tree Sky Rope Athlete Snow Tree Sky Snowboard Large # of uninitialized images Visual Text C Nr O R NF X Ar Nt Z S T Learning Model Recognition & Experiment Dataset Learned Model Results Outline Small # of automatically initialized images

Badminton Bocce Croquet Polo 33 8 Event/Scene Classes Remark: Tags are not used during testing

Rock climbing Rowing Sailing Snow boarding 34 8 Event/Scene Classes

35 C Nr R NFNF X ArAr Nt Z S T Learned model: O D O

36 Athlete Grass Horse C Nr O NFNF X ArAr Nt Z S T D R Learned model: R

37 C Nr O R NFNF X ArAr Nt Z T D S Learned model: S

38 8 way classification: 54% ClassificationAnnotationSegmentation

39 ClassificationAnnotationSegmentation Alipr: Li et al 03Corr LDA: Blei et al 03

40 ClassificationAnnotationSegmentation

Effect of top-down class context 41 Horse C O R X Z T S O R X Z T S Model w/o top-down classFull Model

Athlete Horse Grass Tree Saddle Wind Athlete Rock Grass Tree Sky Rope Athlete Snow Tree Sky Snowboard Large # of uninitialized images Small # of automatically initialized images Visual Text C Nr O R NF X Ar Nt Z S T Sky Athlete Tree Mountain Rock Class: Rock climbing Athlete Mountain Tree Rock Sky Ascent Sky Athlete Water Tree sailboat Class: Sailing Athlete Sailboat Tree Water Sky Wind LearningModel Recognition & Experiment Tree Athlete Snowboard Snow Class: Snowboarding Athlete Snowboard Tree Snow Sky Powder

Thank Prof. Silvio Savarese, Juan Carlos Niebles, Chong Wang, Barry Chai, Min Sun, Bangpeng Yao, Hao Su, Jia Deng, anonymous reviewers And You 43