1. CS 664, Session 20 1 Minimal Subscene Working definition: The smallest set of objects, actors and actions in a dynamic visual scene that are relevant to present behavior For now we will assume: Bottom-up: objects/actors/actions must be visible Top-down: relevance to present behavior explicitly specified, e.g., by specifying a question or task Knowledge base: the system may supplement explicit knowledge with long-term acquired knowledge

2. CS 664, Session 20 2 General architecture

3. CS 664, Session 20 3 Factors influencing selection of minimal subscene At least include… Setting/gist/layout Bottom-up salience Cultural/learned Top-down

4. CS 664, Session 20 4 Factors influencing selection of minimal subscene (1) Setting/gist/layout: selected objects/actors/actions tended to be: - at center of field of view - in the foreground / occludes other objects/actors/actions - followed by camera if camera moved - present throughout video clip - often getting closer / growing larger E.g., boy playing with scooter; bare-chested man standing & drinking Caveats: - lack of stereo increases foreground/background interferences - having everything in focus is unnatural - ambiguity in selection of minimal subscene if actors pass by - selected minimal subscene may disintegrate due to occlusions

5. CS 664, Session 20 5 Factors influencing selection of minimal subscene (2) Bottom-up salience: introspection as well as model suggest that selected objects/actors/actions were fairly salient E.g., boy riding scooter; bare-chested man Note: motion cues widely agreed to be the strongest Caveats: - low-quality video makes details difficult to perceive - salient distracting actors may disengage attention from current minimal subscene

6. CS 664, Session 20 6 Factors influencing selection of minimal subscene (3) Cultural/learned: some actors/objects/actions may bear culturally strong meaning that is likely to make them belong to the minimal subscene E.g., finger pointing movement; facial expressions; alpha male Caveats: - culture-specific (? – not tested) - experience-specific (? – not tested)

7. CS 664, Session 20 7 Factors influencing selection of minimal subscene (4) Top-down: behavioral priorities and personal likings influence selection of components of minimal subscene E.g., nerd playing with electronic gadget; handsome man; pretty girl; groups more interesting than isolated people Caveats: - somewhat linked to cultural - gender-specific differences - most probably influenced by nature of task but we have not explicitly tested for that

8. CS 664, Session 20 8 Nature of minimal subscene Exploration mode: Initial selection seems guided by setting/gist/layout as well as salience E.g., focus on salient actors at center & foreground Analysis mode: once locked onto a minimal subscene, all background activity becomes distracting Disengagement: if background distractor strong enough, may break current minimal subscene and trigger analysis of another minimal subscene E.g., nice girl passing by

9. CS 664, Session 20 9 Additional caveats If a minimal subscene is too boring, it will easily disintegrate E.g., second clip with boy & dad playing with scooter: “pffff, him again, I know what he will be doing, so let’s check out what else is happening” Cameraman may have strong influence on which minimal subscene is selected E.g., by determining centering, by following some actors (or not following them – which may be perceived as unnatural and be distracting) In extended video clips, several minimal subscenes may be selected in sequence E.g., first boy with scooter, then pretty girl, then man with dog, etc.

10. CS 664, Session 20 10 Can we deal with all that? Setting/gist/layout: in principle, yes – some limited models exist Bottom-up salience: should be fine based on previous modeling Cultural/learned: very difficult for a computer system! Cues often very subtle (e.g., facial expressions) or involve complex spatial transformations (e.g., pointing to a location in 3D space) Top-down: should be fine based on previous modeling work

11. CS 664, Session 20 11 General architecture It is important to Note that the General architecture Seems to support all Functions just described.

12. CS 664, Session 20 12 More video clips? Multi-threaded events / interactions Influence of task Foreground/background ambiguities Cross-clip continuity Effects of scale Multiple simultaneous subscenes Etc…

13. CS 664, Session 20 13 Examples / experiments Examine video clips For each scene, please write down: Most salient object Most salient action Minimal subscene Who is doing what to whom

14. CS 664, Session 20 14 Scene 018

15. CS 664, Session 20 15 Scene 018 – Attentional Trajectory

16. CS 664, Session 20 16 Scene 019

17. CS 664, Session 20 17 Scene 019 – Attentional Trajectory

18. CS 664, Session 20 18 Scene 020

19. CS 664, Session 20 19 Scene 020 – Attentional Trajectory

20. CS 664, Session 20 20 Scene 021

21. CS 664, Session 20 21 Scene 021 – Attentional Trajectory

22. CS 664, Session 20 22 Scene 022

23. CS 664, Session 20 23 Scene 022 – Attentional Trajectory

24. CS 664, Session 20 24 Scene 023

25. CS 664, Session 20 25 Scene 023 – Attentional Trajectory

26. CS 664, Session 20 26 Scene 024

27. CS 664, Session 20 27 Scene 024 – Attentional Trajectory

28. CS 664, Session 20 28 Scene 025

29. CS 664, Session 20 29 Scene 025 – Attentional Trajectory

30. CS 664, Session 20 30 Scene 026

31. CS 664, Session 20 31 Scene 026 – Attentional Trajectory

32. CS 664, Session 20 32 Scene 027

33. CS 664, Session 20 33 Scene 027 – Attentional Trajectory

34. CS 664, Session 20 34 Scene 028

35. CS 664, Session 20 35 Scene 028 – Attentional Trajectory

36. CS 664, Session 20 36 Scene 029

37. CS 664, Session 20 37 Scene 029 – Attentional Trajectory

38. CS 664, Session 20 38 Scene 030

39. CS 664, Session 20 39 Scene 030 – Attentional Trajectory

40. CS 664, Session 20 40 Scene 031

41. CS 664, Session 20 41 Scene 031 – Attentional Trajectory

42. CS 664, Session 20 42 Scene 032

43. CS 664, Session 20 43 Scene 032 – Attentional Trajectory