1. C ONSTRAINED S EMI -S UPERVISED L EARNING USING A TTRIBUTES AND C OMPARATIVE A TTRIBUTES Presenter : Ankit Laddha Most of the slides are borrowed from Abhinav Shrivastava’s ECCV talk

2. O UTLINE Supervision based problem definitions Bootstrapping Approach Experimental Evaluation 2

3. S UPERVISION B ASED P ROBLEM D EFINITIONS A CTIVE L EARNING B IG -D ATA 3 [von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009] [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]

4. A CTIVE LEARNING U NSUPERVISED 4 [Russell et al., CVPR 2006] [Kang et al., ICCV 2011] S UPERVISION B ASED P ROBLEM D EFINITIONS

5. A CTIVE L EARNING U NSUPERVISED S EMI -S UPERVISED 5 [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009] S UPERVISION B ASED P ROBLEM D EFINITIONS

6. 6 B OOTSTRAPPING

7. Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Train Models Add to Labeled Set Retrain Models Amphitheatre 7

8. B OOTSTRAPPING - P ROBLEMS Retrain Models Labeled Seed Examples Amphitheatre Unlabeled Data Select Candidates Add to Labeled Set Amphitheatre 25 th Iteration 8 [Curran et al., PACL 2007] Semantic Drift Amphitheatre + Auditorium

9. 9 A PPROACH

10. 10 A PPROACH

11. M UTUAL E XCLUSION (ME) 11

12. Amphitheatre Auditorium Banquet Hall Banquet Hall Conference Room Conference Room B INARY A TTRIBUTES (BA) 12 [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

13. B INARY A TTRIBUTES (BA) Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made [Patterson and Hays, CVPR 2012] Tables and Chairs Conference Room Banquet Hall Auditorium Amphitheatre Indoor Large Seating Capacity Man-made

14. Auditorium 14

15. S HARING VIA D ISSIMILARITY 15 AmphitheatreAuditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] C OMPARATIVE A TTRIBUTES

16. AmphitheatreAuditorium ? 16

17. 17 AmphitheatreAuditorium

18. D ISSIMILARITY 18 C OMPARATIVE A TTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Features GIST RGB (Tiny Image) Line Histogram of:  Length  Orientation LAB histogram [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

19. ………… D ISSIMILARITY 19 C OMPARATIVE A TTRIBUTES [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ………… Has Larger Circular Structures Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] or Has Larger Circular Structures

20. C OMPARATIVE A TTRIBUTES 20 [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] Amphitheatre>Barn Amphitheatre>Conference Room Desert>Barn Is More Open Church (Outdoor)>Cemetery Barn>Cemetery Has Taller Structures

21. Conference Room Iteration 1 Iteration 40 Seed Examples 21 Introspection

22. 22 Unary Binary

23. Banquet Bedroom Labeled Data Unlabeled Data has more space has larger structures Training Pairwise Data Promoted Instances Conference Room Banquet Hall 23 [Gupta and Davis, ECCV 2008] Comparative Attribute Classifiers more space larger structures Attribute Classifiers indoor has grass Scene Classifiers bedroom banquet hall

24. E XPERIMENTAL E VALUATION 24

25. E XPERIMENTAL S ETUP Experiments Control Small-scale Large-scale Evaluation Metrics Scene Classification – Mean AP Purity of Labels Datasets SUN Database ImageNet 15 Scene Categories 25

26. C ONTROL E XPERIMENTS 26 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

27. C ONTROL E XPERIMENTS 27

28. C ONTROL E XPERIMENTS 28

29. C ONTROL E XPERIMENTS 29

30. C ONTROL E XPERIMENTS 30

31. C ONTROL E XPERIMENTS 31 \\

32. Bootstrapping BA Constraints Amphitheatre Our Approach Seed Images 32

33. C O - TRAINING (S MALL S CALE ) 33 # Images (SUN Database) 15 Scene Classes2 (seed) Black-box Binary Attributes (BA) 15x2 (seed) Black-box Comparative Attributes (CA) 15x2 (seed) Unlabeled Dataset18,000 (Distractors)(9,500) Test Set50 SUN Database : [Xiao et al., CVPR 2010]

34. B ASELINES Bootstrapping (Self-learning) -Binary Classifiers -Multi-class Classifiers Eigen Function based Graph Laplacian 34

35. S CENE C LASSIFICATION 35 Eigen Functions: [Fergus et al., NIPS 2009]

36. P URITY OF L ABELING 36 Eigen Functions: [Fergus et al., NIPS 2009]

37. Iteration-1 Iteration-60 Bootstrapping Our Approach Iteration-1 Iteration-60 Seed Images Bedroom 37

38. 1 40 Banquet Hall 10 Iterations Seed Images 38

39. C O - TRAINING ( LARGE S CALE ) 15 Scene Categories  25 Seed images / category Unlabeled Set  1Million (SUN Database + ImageNet)  >95% distractors 39 SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009] Improve 12 out of 15 scene classifiers

40. 40

41. S UPERVISION S UPERVISED U NSUPERVISED A CTIVE L EARNING S EMI -S UPERVISED G RAPH - BASED M ETHODS 41 [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

42. S CENE C LASSIFICATION - C ONTROL 42 Eigen Functions: [Fergus et al., NIPS 2009]

43. P URITY OF L ABELING - C ONTROL 43 Eigen Functions: [Fergus et al., NIPS 2009]

44. E IGEN F UNCTIONS 44

45. F EATURES 960D GIST 75D RGB (image is resized to 5x5) 30D histogram of line lengths 200D histogram of orientation of lines 784D 3D-histogram Lab color space (14x14x4) Total 2049 45 [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

46. C ATEGORIES 46

47. B INARY A TTRIBUTE R ELATIONSHIP 47

48. C OMPARATIVE A TTRIBUTE R ELATIONSHIPS 48

49. C LASSIFIERS Boosted Decision Trees – From [Hoiem et al., IJCV 2007] Scene – 20 Trees, 8 Nodes Binary Attribute – 40 Trees, 8 Nodes Comparative Attribute 20 Trees, 4 Nodes Differential features as in [Gupta and Davis, ECCV 2008] 49

50. 50

51. 1 40 Banquet Hall 90 10 Iterations Seed Images 51

52. 52

53. S CENE C LASSIFICATION 53 Mean

54. Amphitheatre Auditorium Amphitheatre Auditorium Labeled Seed Examples Bootstrapping Selected Candidates 54

55. Labeled Seed Examples Amphitheatre Auditorium Amphitheatre Auditorium Bootstrapping Amphitheatre Auditorium Our Approach (Constrained Bootstrapping) Selected Candidates Indoor Has Seat Rows Attributes Has Larger Circular Structures Comparative Attributes 55