Download presentation
Presentation is loading. Please wait.
Published byBlaise Anthony Modified over 9 years ago
1
Sung Ju Hwang and Kristen Grauman University of Texas at Austin
2
Images tagged with keywords clearly tell us which objects to search for Detecting tagged objects Dog Black lab Jasper Sofa Self Living room Fedora Explore #24
3
Duygulu et al. 2002 Detecting tagged objects Previous work using tagged images focuses on the noun ↔ object correspondence. Fergus et al. 2005 Li et al., 2009 Berg et al. 2004 [Lavrenko et al. 2003, Monay & Gatica-Perez 2003, Barnard et al. 2004, Schroff et al. 2007, Gupta & Davis 2008, Vijayanarasimhan & Grauman 2008, …]
4
Mug Key Keyboard Toothbrush Pen Photo Post-it Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster ?? Based on tags alone, can you guess where and what size the mug will be in each image? Our Idea The list of human-provided tags gives useful cues beyond just which objects are present.
5
Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster Mug Key Keyboard Toothbrush Pen Photo Post-it Our Idea Presence of larger objects Mug is named first Absence of larger objects Mug is named later The list of human-provided tags gives useful cues beyond just which objects are present.
6
Our Idea We propose to learn the implicit localization cues provided by tag lists to improve object detection.
7
Woman Table Mug Ladder Approach overview Mug Key Keyboard Toothbrush Pen Photo Post-it Object detector Implicit tag features Computer Poster Desk Screen Mug Poster Training: Learn object-specific connection between localization parameters and implicit tag features. Mug Eiffel Desk Mug Office Mug Coffee Testing: Given novel image, localize objects based on both tags and appearance. P (location, scale | tags) Implicit tag features
8
Woman Table Mug Ladder Approach overview Mug Key Keyboard Toothbrush Pen Photo Post-it Object detector Implicit tag features Computer Poster Desk Screen Mug Poster Training: Learn object-specific connection between localization parameters and implicit tag features. Mug Eiffel Desk Mug Office Mug Coffee Testing: Given novel image, localize objects based on both tags and appearance. P (location, scale | tags) Implicit tag features
9
Feature: Word presence/absence Mug Key Keyboard Toothbrush Pen Photo Post-it Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster Presence or absence of other objects affects the scene layout record bag-of-words frequency. Presence or absence of other objects affects the scene layout = count of i-th word., where MugPenPost-it Tooth brush KeyPhotoComputerScreenKeyboardDeskBookshelfPoster W(im1)111111001000 W(im2)100000121111
10
Feature: Word presence/absence Mug Key Keyboard Toothbrush Pen Photo Post-it Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster Presence or absence of other objects affects the scene layout record bag-of-words frequency. Presence or absence of other objects affects the scene layout Large objects mentionedSmall objects mentioned = count of i-th word., where MugPenPost-it Tooth brush KeyPhotoComputerScreenKeyboardDeskBookshelfPoster W(im1)111111001000 W(im2)100000121111
11
Feature: Rank of tags People tag the “important” objects earlierPeople tag the “important” objects earlier record rank of each tag compared to its typical rank. = percentile rank of i-th word., where Mug Key Keyboard Toothbrush Pen Photo Post-it Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster MugComputerScreenKeyboardDeskBookshelfPosterPhotoPenPost-it Tooth brush Key R(im1) 0.80000.510000.280.720.8200.90 R(im2) 0.230.620.210.130.480.610.4100000
12
Feature: Rank of tags People tag the “important” objects earlierPeople tag the “important” objects earlier record rank of each tag compared to its typical rank. = percentile rank of i-th word., where Mug Key Keyboard Toothbrush Pen Photo Post-it Computer Poster Desk Bookshelf Screen Keyboard Screen Mug Poster Relatively high rank MugComputerScreenKeyboardDeskBookshelfPosterPhotoPenPost-it Tooth brush Key R(im1) 0.80000.510000.280.720.8200.90 R(im2) 0.230.620.210.130.480.610.4100000
13
Feature: Proximity of tags People tend to move eyes to nearby objects after first fixation record proximity of all tag pairs. People tend to move eyes to nearby objects after first fixation = rank difference., where 1) Mug 2) Key 3) Keyboard 4) Toothbrush 5) Pen 6) Photo 7) Post-it 1) Computer 2) Poster 3) Desk 4) Bookshelf 5) Screen 6) Keyboard 7) Screen 8) Mug 9) Poster 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 MugScreenKeyboardDeskBookshelf Mug100.500 Screen 0000 Keyboard 100 Desk 00 Bookshelf 0 MugScreenKeyboardDeskBookshelf Mug110.50.20.25 Screen 110.330.5 Keyboard 10.330.5 Desk 11 Bookshelf 1
14
Feature: Proximity of tags People tend to move eyes to nearby objects after first fixation record proximity of all tag pairs. People tend to move eyes to nearby objects after first fixation = rank difference., where 1) Mug 2) Key 3) Keyboard 4) Toothbrush 5) Pen 6) Photo 7) Post-it 1) Computer 2) Poster 3) Desk 4) Bookshelf 5) Screen 6) Keyboard 7) Screen 8) Mug 9) Poster 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 MugScreenKeyboardDeskBookshelf Mug100.500 Screen 0000 Keyboard 100 Desk 00 Bookshelf 0 MugScreenKeyboardDeskBookshelf Mug110.50.20.25 Screen 110.330.5 Keyboard 10.330.5 Desk 11 Bookshelf 1 May be close to each other
15
Woman Table Mug Ladder Approach overview Mug Key Keyboard Toothbrush Pen Photo Post-it Object detector Implicit tag features Computer Poster Desk Screen Mug Poster Mug Eiffel Desk Mug Office Mug Coffee P (location, scale | W,R,P) Implicit tag features Training: Testing:
16
Modeling P(X|T) We model it directly using a mixture density network (MDN) [Bishop, 1994]. We need PDF for location and scale of the target object, given the tag feature: P(X = scale, x, y | T = tag feature) Input tag feature (Words, Rank, or Proximity) Mixture model Neural network αµ Σ αµ Σ αµ Σ
17
Lamp Car Wheel Light Window House Car Road House Lightpole Car Windows Building Man Barrel Car Truck Car Boulder Car Modeling P(X|T) Example: Top 30 most likely localization parameters sampled for the object “car”, given only the tags.
18
Lamp Car Wheel Light Window House Car Road House Lightpole Car Windows Building Man Barrel Car Truck Car Boulder Car Modeling P(X|T) Example: Top 30 most likely localization parameters sampled for the object “car”, given only the tags.
19
Woman Table Mug Ladder Approach overview Mug Key Keyboard Toothbrush Pen Photo Post-it Object detector Implicit tag features Computer Poster Desk Screen Mug Poster Mug Eiffel Desk Mug Office Mug Coffee P (location, scale | W,R,P) Implicit tag features Training: Testing:
20
Integrating with object detector How to exploit this learned distribution P(X|T)? 1)Use it to speed up the detection process (location priming)
21
Integrating with object detector How to exploit this learned distribution P(X|T)? 1)Use it to speed up the detection process (location priming) (a) Sort all candidate windows according to P(X|T). Most likely Less likely Least likely (b) Run detector only at the most probable locations and scales.
22
Integrating with object detector How to exploit this learned distribution P(X|T)? 1)Use it to speed up the detection process (location priming) 2)Use it to increase detection accuracy (modulate the detector output scores) Predictions from object detector 0.7 0.8 0.9 Predictions based on tag features 0.3 0.2 0.9
23
Integrating with object detector How to exploit this learned distribution P(X|T)? 1)Use it to speed up the detection process (location priming) 2)Use it to increase detection accuracy (modulate the detector output scores) 0.63 0.24 0.18
24
Experiments: Datasets LabelMePASCAL VOC 2007 Street and office scenes Contains ordered tag lists via labels added 5 classes 56 unique taggers 23 tags / image Dalal & Trigg’s HOG detector Flickr images Tag lists obtained on Mechanical Turk 20 classes 758 unique taggers 5.5 tags / image Felzenszwalb et al.’s LSVM detector
25
Experiments We evaluate Detection Speed Detection Accuracy We compare Raw detector (HOG, LSVM) Raw detector + Our tag features We also show the results when using Gist [Torralba 2003] as context, for reference.
26
We search fewer windows to achieve same detection rate. We know which detection hypotheses to trust most. PASCAL: Performance evaluation Naïve sliding window searches 70%. We search only 30%.
27
PASCAL: Accuracy vs Gist per class
29
Lamp Person Bottle Dog Sofa Painting Table Person Table Chair Mirror Tablecloth Bowl Bottle Shelf Painting Food Bottle Car License Plate Building Car LSVM+ Tags (Ours) LSVM alone PASCAL: Example detections Car Door Gear Steering Wheel Seat Person Camera
30
Dog Floor Hairclip Dog Person Ground Bench Scarf Person Microphone Light Horse Person Tree House Building Ground Hurdle Fence PASCAL: Example detections Dog Person LSVM+Tags (Ours)LSVM alone
31
Aeroplane Sky Building Shadow Person Pole Building Sidewalk Grass Road Dog Clothes Rope Plant Ground Shadow String Wall Bottle Glass Wine Table PASCAL: Example failure cases LSVM+Tags (Ours)LSVM alone
32
Results: Observations Often our implicit features predict: - scale well for indoor objects - position well for outdoor objects Gist usually better for y position, while our tags are generally stronger for scale Need to have learned about target objects in variety of examples with different contexts - visual and tag context are complementary
33
Summary We want to learn what is implied (beyond objects present) by how a human provides tags for an image. Approach translates existing insights about human viewing behavior (attention, importance, gaze, etc.) into enhanced object detection. Novel tag cues enable effective localization prior. Significant gains with state-of-the-art detectors and two datasets.
34
Joint multi-object detection From tags to natural language sentences Image retrieval applications Future work
35
Summary We want to learn what is implied (beyond objects present) by how a human provides tags for an image. Approach translates existing insights about human viewing behavior (attention, importance, gaze, etc.) into enhanced object detection. Novel tag cues enable effective localization prior. Significant gains with state-of-the-art detectors and two datasets.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.