1. Scalable Multi-Label Annotation Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei

2. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ++---- Task: Crowdsource object labels for images. Generalization: musical attributes of songs actions in movies sentiments in documents Application: Benchmarking, training, modeling Multi-label annotation Data Item Labels

3. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Current focus: 200 Category Detection (~100,000 fully labeled images) Large-Scale Visual Recognition Challenge ILSVRC 2010-2014

4. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ?????? Answer Question MachineCrowd Is there a table? Yes

5. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird +????? Answer Question MachineCrowd Is there a table? Yes

6. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++???? Answer Question MachineCrowd Is there a chair? Yes

7. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++-??? Answer Question MachineCrowd Is there a horse? No

8. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++--?? Answer Question MachineCrowd Is there a dog? No

9. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++---? Answer Question MachineCrowd Is there a cat? No

10. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++---- Answer Question MachineCrowd Is there a bird? No

11. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ++---- +---+- ++---- Naïve approach: ask for each object Cost: O(NK) for N images and K objects

12. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ++---- Furniture Mammal Animal Hierarchy

13. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ++---- +---+- ++---- Sparsity Correlation Furniture Mammal Animal Hierarchy

14. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ?????? Answer Question Machine Crowd Furniture Mammal Animal Better approach: exploit label structure

15. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ?????? Answer Question Machine Crowd Is there an animal? No Furniture Mammal Animal Better approach: exploit label structure

16. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question Machine Crowd Is there an animal? No Furniture Mammal Animal Better approach: exploit label structure

17. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question Machine Crowd Is there furniture? Yes Mammal Better approach: exploit label structure Animal Furniture

18. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Answer Question Machine Crowd Is there a table? Yes Mammal Better approach: exploit label structure Animal TableChairHorseDogCatBird ??---- Furniture

19. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird +?---- Answer Question Machine Crowd Is there a chair? Yes Mammal Better approach: exploit label structure Animal Furniture

20. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Answer Question Machine Crowd Is there a chair? Yes Mammal Better approach: exploit label structure Animal TableChairHorseDogCatBird ++---- Furniture

21. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Selecting the Right Question Goal: Get as much utility (new labels) as possible, for as little cost (worker time) as possible, given a desired level of accuracy

22. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Accuracy constraint User-specified accuracy threshold, e.g., 95% Majority voting assuming uniform worker quality [GAL: Sheng, Provost, Ipeirotis KDD ‘08] Might require only one worker, might require several based on the task

23. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Cost: worker time (time = money) Question (is there …)Cost (second) a thing used to open cans/bottles14.4 an item that runs on electricity (plugged in or using batteries) 12.6 a stringed instrument3.4 a canine2.0 expected human time to get an answer with 95% accuracy

24. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Utility: expected # of new labels TableChairHorseDogCatBird ?????? Is there a table? Yes No TableChairHorseDogCatBird +????? TableChairHorseDogCatBird -????? utility = 1

25. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Utility: expected # of new labels TableChairHorseDogCatBird ?????? Is there a table? Yes No TableChairHorseDogCatBird +????? TableChairHorseDogCatBird -????? TableChairHorseDogCatBird ?????? Is there an animal? TableChairHorseDogCatBird ?????? TableChairHorseDogCatBird ??---- utility = 1 utility = 0.5 * 0 + 0.5 * 4 = 2 Pr(Y) = 0.5 Pr(N) = 0.5

26. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Pick the question with the most labels per second Query: Is there a... Utility (num labels) Cost (worker time in secs) Utility-Cost Ratio (labels per sec) mammal with claws or fingers 12.03.04.0 living organism24.87.93.1 mammal17.67.42.4 creature without legs5.92.62.3 land or avian creature20.89.52.2 Selecting the Right Question

27. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Dataset: 20K images from ImageNet Challenge 2013. Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy Results

28. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Dataset: 20K images from ImageNet Challenge 2013. Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy Setup: – 50-50 training test split – Estimate parameters on training, simulate on test – Future work: online estimation Results

29. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: accuracy Accuracy Threshold per question (parameter) Accuracy (F1 score) Naïve approach Accuracy (F1 score) Our approach 0.9599.64 (75.67)99.75 (76.97) 0.9099.29 (60.17)99.62 (60.69) Annotating 10K images with 200 objects

30. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: cost Accuracy Threshold per question (parameter) Cost saving (our approach compared to naïve approach) 0.953.93x 0.906.18x Annotating 10K images with 200 objects

31. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: cost Accuracy Threshold per question (parameter) Cost saving (our approach compared to naïve approach) 0.953.93x 0.906.18x Annotating 10K images with 200 objects 6 times more labels per second

32. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Speeds up crowdsourced multi-label annotation by exploiting the structure and distribution of labels. Could be a bargain for you! TableChairHorseDogCatBird ++---- +---+- ++---- Furniture Mamma l Animal Hierarchy Correlation Sparsity Conclusions

33. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei

34. TableChairHorseDogCatBird ?????? Answer Question MachineCrowd Is there an animal? No Furniture Mammal Animal Better approach: exploiting hierarchy, sparsity and correlation

35. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question MachineCrowd Is there an animal? No Furniture Mammal Animal Better Approach

36. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question MachineCrowd Is there furniture? Yes Furniture Mammal Animal Better Approach

37. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird +?---- Answer Question MachineCrowd Is there a table? Yes Furniture Mammal Animal Better Approach

38. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ++---- Answer Question MachineCrowd Is there a chair? Yes Furniture Mammal Animal Better Approach

39. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Algorithm Initialize all labels to ‘missing’ While there are missing labels do – Select a question Q from all possible questions – Obtain an answer A to question Q from the crowd – Set values of some labels to +1 or -1

40. Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Algorithm Initialize all labels to ‘missing’ While there are missing labels do – Select a question Q from all possible questions – Obtain an answer A to question Q from the crowd – Set values of some labels to +1 or -1