1. Challenges to image parsing researchers Lana Lazebnik UNC Chapel Hill sky sidewalk building road car person car mountain

2. The past: “closed universe” datasets Tens of classes, hundreds of images, offline learning He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs (2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc. Figure from Shotton et al. (2009)

3. Evolving images, annotations http://labelme.csail.mit.edu/ The future: “open universe” datasets

4. Non-uniform class frequencies The future: “open universe” datasets

5.  Combination of local cues?  Multiple segmentations/grouping hypotheses?  Context?  Graphical models (MRFs, CRFs, etc.)?  Offline learning and inference? Which “closed universe” techniques can survive in the “open universe” setting?

6. Learning from all of LabelMe 50K images, 232 labels sky tree road car sky sea sun building window door road sky building mountain sky building sidewalk car road car ceiling wall floor Tighe & Lazebnik, work in progress

7. Learning from all of LabelMe 50K images, 232 labels Per-class classification rates Tighe & Lazebnik, work in progress

8. Challenge: Parsing high-res images

9. Challenge: Dynamic image interpretation  Image parsing algorithms should become autonomous decision-making agents Visual “detective task”: Where was this photo taken?

10. Challenge: Dynamic image interpretation  Image parsing algorithms should become autonomous decision-making agents Input

11. Summary  Challenges to image parsing researchers:  Learn to parse images from “open universe” evolving datasets  Try parsing gigapixel images!  Develop active, sequential image interpretation strategies