C ONSTRAINED S EMI -S UPERVISED L EARNING USING A TTRIBUTES AND C OMPARATIVE A TTRIBUTES Presenter : Ankit Laddha Most of the slides are borrowed from Abhinav Shrivastava’s ECCV talk

O UTLINE Supervision based problem definitions Bootstrapping Approach Experimental Evaluation 2

S UPERVISION B ASED P ROBLEM D EFINITIONS A CTIVE L EARNING B IG -D ATA 3 [von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009] [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]

A CTIVE LEARNING U NSUPERVISED 4 [Russell et al., CVPR 2006] [Kang et al., ICCV 2011] S UPERVISION B ASED P ROBLEM D EFINITIONS

A CTIVE L EARNING U NSUPERVISED S EMI -S UPERVISED 5 [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009] S UPERVISION B ASED P ROBLEM D EFINITIONS

6 B OOTSTRAPPING

Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Train Models Add to Labeled Set Retrain Models Amphitheatre 7

B OOTSTRAPPING - P ROBLEMS Retrain Models Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Add to Labeled Set Amphitheatre 25 th Iteration 8 [Curran et al., PACL 2007] Semantic Drift Amphitheatre + Auditorium

9 A PPROACH

10 A PPROACH

M UTUAL E XCLUSION (ME) 11

Amphitheatre Auditorium Banquet Hall Banquet Hall Conference Room Conference Room B INARY A TTRIBUTES (BA) 12 [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

B INARY A TTRIBUTES (BA) Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made [Patterson and Hays, CVPR 2012] Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made

Auditorium 14

S HARING VIA D ISSIMILARITY 15 AmphitheatreAuditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] C OMPARATIVE A TTRIBUTES

AmphitheatreAuditorium ? 16

17 AmphitheatreAuditorium

D ISSIMILARITY 18 C OMPARATIVE A TTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Features GIST RGB (Tiny Image) Line Histogram of:  Length  Orientation LAB histogram [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

………… D ISSIMILARITY 19 C OMPARATIVE A TTRIBUTES [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Has Larger Circular Structures Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] or Has Larger Circular Structures

C OMPARATIVE A TTRIBUTES 20 [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] Amphitheatre>Barn Amphitheatre>Conference Room Desert>Barn Is More Open Church (Outdoor)>Cemetery Barn>Cemetery Has Taller Structures

Conference Room Iteration 1 Iteration 40 Seed Examples 21 Introspection

22 Unary Binary

Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall 23 [Gupta and Davis, ECCV 2008] Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

E XPERIMENTAL E VALUATION 24

E XPERIMENTAL S ETUP Experiments Control Small-scale Large-scale Evaluation Metrics Scene Classification – Mean AP Purity of Labels Datasets SUN Database ImageNet 15 Scene Categories 25

C ONTROL E XPERIMENTS 26 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

C ONTROL E XPERIMENTS 27

C ONTROL E XPERIMENTS 28

C ONTROL E XPERIMENTS 29

C ONTROL E XPERIMENTS 30

C ONTROL E XPERIMENTS 31 \\

Bootstrapping BA Constraints Amphitheatre Our Approach Seed Images 32

C O - TRAINING (S MALL S CALE ) 33 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 15x2 (seed) Black-box Comparative Attributes (CA) 15x2 (seed) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

B ASELINES Bootstrapping (Self-learning) -Binary Classifiers -Multi-class Classifiers Eigen Function based Graph Laplacian 34

S CENE C LASSIFICATION 35 Eigen Functions: [Fergus et al., NIPS 2009]

P URITY OF L ABELING 36 Eigen Functions: [Fergus et al., NIPS 2009]

Iteration-1 Iteration-60 Bootstrapping Our Approach Iteration-1 Iteration-60 Seed Images Bedroom 37

1 40 Banquet Hall 10 Iterations Seed Images 38

C O - TRAINING ( LARGE S CALE ) 15 Scene Categories  25 Seed images / category Unlabeled Set  1Million (SUN Database + ImageNet)  >95% distractors 39 SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009] Improve 12 out of 15 scene classifiers

40

S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS 41 [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

S CENE C LASSIFICATION - C ONTROL 42 Eigen Functions: [Fergus et al., NIPS 2009]

P URITY OF L ABELING - C ONTROL 43 Eigen Functions: [Fergus et al., NIPS 2009]

E IGEN F UNCTIONS 44

F EATURES 960D GIST 75D RGB (image is resized to 5x5) 30D histogram of line lengths 200D histogram of orientation of lines 784D 3D-histogram Lab color space (14x14x4) Total [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

C ATEGORIES 46

B INARY A TTRIBUTE R ELATIONSHIP 47

C OMPARATIVE A TTRIBUTE R ELATIONSHIPS 48

C LASSIFIERS Boosted Decision Trees – From [Hoiem et al., IJCV 2007] Scene – 20 Trees, 8 Nodes Binary Attribute – 40 Trees, 8 Nodes Comparative Attribute 20 Trees, 4 Nodes Differential features as in [Gupta and Davis, ECCV 2008] 49

50

1 40 Banquet Hall Iterations Seed Images 51

52

S CENE C LASSIFICATION 53 Mean

Amphitheatre Auditorium Amphitheatre Auditorium Labeled Seed Examples Bootstrapping Selected Candidates 54

Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Our Approach (Constrained Bootstrapping) Selected Candidates Indoor Has Seat Rows Attributes Has Larger Circular Structures Comparative Attributes 55