1. Beyond Nouns Exploiting Preposition and Comparative adjectives for learning visual classifiers

2. Nouns Nouns Images Widely used Object detection Tag recommendation etc Co-occurrence over large data -> relations

3. Co-occurrence Examples Jack and Jill Laurel and Hardy

4. Co-occurrence

5. Co-occurrence Co-occurrence is good To detect whether the objects are there Eg. Suggesting tags Co-occurrence is bad When we want to disambiguate We might never be able to tell Jack and Jill apart

6. Relationships How do they help? Additional constraints limit possibilities Other intuitive examples Aptitude questions Like ILP? “Since cars are typically found on streets, it is difficult to resolve the correspondence using co- occurrence alone”

7. Relationships

8. Relationships How do we learn them? Manually annotate Learn model Use relationship to predict in testing …. Or …

9. The Problem We have a weakly labeled dataset (tags only) Relationship model helps us label it strongly Strong labeling helps us derive relationship model Therefore, EM. The labeling (assignment) is treated as the missing data.

10. Feature Representation

11. Model

12. EM

13. EM M For the noun assignment done earlier, we learn relationship and object classifiers The relationship classifier is modeled on a single feature … GOTO E Boot strapping can be done using any image annotation approach

14. Model

15. Inference

16. Inference

17. Testing Likelihood models Nouns: Nearest Neighbor Relationships: Decision Stump Evaluation on: subset of Corel5k Training on: 850 images with (173) nouns and (19) hand labeled relationships

18. Evaluation – resolution of correspondence ambiguities Metrics range of semantics number of unique nouns correct (?) frequency correct number of total nouns correct (?) Compared with Image annotation algorithms Human assisted annotation (are machines better at relationships than us?)

19. Results

20. Examples

21. Evaluation – labeling new images Tested on random subset of Corel5k, based on learnt vocabulary Labels verified by “us” (Corel is misleading) Precision/Recall While recall rates are reported with respect to corel annotations, precision rates are reported with respect to correctness defined by human observers.

22. Results

23. Results

24. Examples