2. A Visual Stepping Stone to AI
Larry Zitnick
Facebook AI Research

3. Recognition?
Neocognitron, 1983
1984

4. 2016 1984 Recognition Data GPUs Backprop AlexNet, 2012
Neocognitron, 1983
1984

5. AI
2048
Recognition
2016
Data
GPUs
Backprop
1984

6. 2048 2016 1984 AI Recognition More data More compute Backprop Data
GPUs
Backprop
1984

7. A giraffe standing in the grass next to a tree.
A man riding a wave on a surfboard in the water.
Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation, Chen and Zitnick, CVPR 2015.

8. An airplane is parked on the tarmac at an airport.
NYU, MIT, Harvard
A man riding a
motorcycle on a beach.
Building Machines That Learn and Think Like People,
Lake et al., ArXiv , 2016

9. MIRROR

10. 2048 2016 1984 AI Recognition More Data More Compute Backprop Data
GPUs
Backprop
1984

11. Recognition
AI ?
2016
Data
GPUs
Backprop
1984

12. LEARNING

13. I’ll give you a $1 if you find a cat image.
There is a cat in this image.
Supervised
Reinforcement
I’m not telling you anything!!!
I didn’t look closely but I think there’s a cat.
Semi-supervised
Unsupervised

14. Supervised

15. Semi-supervised learning
A yellow Vespa parked in a lot with other cars.
A store display that has a lot of bananas on sale.

16. Unlikely Likely fence pink
Learning Visual Classifiers using Human-centric Annotations Misra et al., CVPR 2016

17. Reinforcement learning
MazeBase:
A sandbox for experimenting with reinforcement learning, reasoning, and planning.
MazeBase: A Sandbox for Learning From Games
Sukhbaatar et al., arXiv, 2016

18. Unsupervised learning
Correct
Wrong
Unsupervised Learning using Sequential Verification for Action Recognition, Misra et al., arXiv, 2016

19. Visual ^
REASONING

20. UNDERSTANDING Mary went to the hallway. John moved to the bathroom.
Mary travelled to the kitchen.
Where is Mary?
Dialog-based Language Learning, Weston, ArXiv , 2016

21. PREDICTING Learning Physical Intuition of Block Towers by Example
Lerer et al., arXiv , 2016

22. DATA
We Are Humor Beings: Understanding and Predicting Visual Humor, Chandrasekaran et al., CVPR 2016

23. Visual Question Answering
MEASURING
Visual Question Answering
Is this a vegetarian pizza?
Does this person have 20/20 vision?
VQA: Visual Question Answering
Antol et al., ICCV 2015

24. Looking forward

25. Recognition
Data
GPUs
Backprop
2016
Learning
Reasoning AI
1984
2048

26. Recognition
Data
GPUs
Backprop
2016
Learning AI
Reasoning
1984
2048

27. Recognition
Learning
Reasoning
2016 AI
1984
20**

28. Facebook AI Research