1. CS 664, Session 19 1 General architecture

2. CS 664, Session 19 2 Minimal Subscene Working definition: The smallest set of objects, actors and actions in a dynamic visual scene that are relevant to present behavior For now we will assume: Bottom-up: objects/actors/actions must be visible Top-down: relevance to present behavior explicitly specified, e.g., by specifying a question or task Knowledge base: the system may supplement explicit knowledge with long-term acquired knowledge

3. CS 664, Session 19 3 Motivation: Humans 1) Free examination 2) estimate material circumstances of family 3) give ages of the people 4) surmise what family has been doing before arrival of “unexpected visitor” 5) remember clothes worn by the people 6) remember position of people and objects 7) estimate how long the “unexpected visitor” has been away from family Yarbus, 1967

4. CS 664, Session 19 4 “Beobot”

5. CS 664, Session 19 5 Visual Attention see http://iLab.usc.edu

6. CS 664, Session 19 6 Object Recognition Riesenhuber & Poggio, Nat Neurosci, 1999 (MIT)

7. CS 664, Session 19 7 Action Recognition Oztop & Arbib, 2001

8. CS 664, Session 19 8 Start: -Issue question -Parse question -Extract keywords -Expand to related concepts, using ontology/KB -Fill initial “task list”

9. CS 664, Session 19 9 Task list Working list of currently relevant objects/actors/actions -Initially empty -Question/task specification provides initial filling-in -As the scene is scanned and objects/actors/actions are recognized, contents of task list are updated

10. CS 664, Session 19 10 “Where:” attention, saliency map and task map Input: video stream Low-level vision: massively parallel extraction of simple visual features from video input Saliency map: localizes conspicuous (potentially interesting) objects irrespectively of why they are salient Task map: acts as spatial filter to saliency map; only locations in the current minimal subscene can easily pass through. Other locations need to be exceptionally salient to pass through.

11. CS 664, Session 19 11 “What” memory Relates concepts to visual properties Bridge between visual and semantic knowledge

12. CS 664, Session 19 12 General architecture

13. CS 664, Session 19 13 Examples / experiments Examine video clips For each scene, please write down: Most salient object Most salient action Minimal subscene Who is doing what to whom

14. CS 664, Session 19 14 Scene 001

15. CS 664, Session 19 15 Scene 001 – Attentional Trajectory

16. CS 664, Session 19 16 Scene 002

17. CS 664, Session 19 17 Scene 002 – Attentional Trajectory

18. CS 664, Session 19 18 Scene 003

19. CS 664, Session 19 19 Scene 003 – Attentional Trajectory

20. CS 664, Session 19 20 Scene 004

21. CS 664, Session 19 21 Scene 004 – Attentional Trajectory

22. CS 664, Session 19 22 Scene 005

23. CS 664, Session 19 23 Scene 005 – Attentional Trajectory

24. CS 664, Session 19 24 Scene 006

25. CS 664, Session 19 25 Scene 006 – Attentional Trajectory

26. CS 664, Session 19 26 Scene 007

27. CS 664, Session 19 27 Scene 007 – Attentional Trajectory

28. CS 664, Session 19 28 Scene 008

29. CS 664, Session 19 29 Scene 008 – Attentional Trajectory

30. CS 664, Session 19 30 Scene 009

31. CS 664, Session 19 31 Scene 009 – Attentional Trajectory

32. CS 664, Session 19 32 Scene 010

33. CS 664, Session 19 33 Scene 010 – Attentional Trajectory

34. CS 664, Session 19 34 Scene 011

35. CS 664, Session 19 35 Scene 011 – Attentional Trajectory

36. CS 664, Session 19 36 Scene 012

37. CS 664, Session 19 37 Scene 012 – Attentional Trajectory

38. CS 664, Session 19 38 Scene 013

39. CS 664, Session 19 39 Scene 013 – Attentional Trajectory

40. CS 664, Session 19 40 Scene 014

41. CS 664, Session 19 41 Scene 014 – Attentional Trajectory

42. CS 664, Session 19 42 Scene 015

43. CS 664, Session 19 43 Scene 015 – Attentional Trajectory

44. CS 664, Session 19 44 Scene 016

45. CS 664, Session 19 45 Scene 016 – Attentional Trajectory

46. CS 664, Session 19 46 Scene 017

47. CS 664, Session 19 47 Scene 017 – Attentional Trajectory