1. Cognitive Computer Vision Kingsley Sage khs20@sussex.ac.uk and Hilary Buxton hilaryb@sussex.ac.uk Prepared under ECVision Specific Action 8-3 http://www.ecvision.org

2. Course outline What is Cognitive Computer Vision (CCV) ? Generative models Graphical models Techniques for modelling cognitive aspects of CCV – Bayesian inference – Markov Models Research issues Coursework and case studies

3. So what is CCV ?

4. In this course, we focus on using of ideas from cognitive science and psychology to do CCV To show how we can build effective CCV systems that are more robust and more capable of solving non-trivial problems than those that do not embrace these ideas Use statistical inference and machine learning as our tools for modelling cognitively inspired processes We are not claiming “hard AI” in this course

5. Key Cognitive Elements Objects, events, activities and behaviours – “What is it that we are observing?” Attention and control – “How is it that we observe?”

6. Key Cognitive Elements Visual learning and memory – Representation of objects and their behaviour – Recognition – Categorisation – These are “what” problems Visual control and attention – Perception for tasks using models of expectation – Goals, task context – Resources, embodiment – These are “how” problems Cognition – From perception to action

7. Key Cognitive Elements Visual learning and memory - examples – Learning about objects and how their appearance can change – Recognising activities by the interactions between objects – Extracting invariant models from training data

8. Learning and “recognising” objects (Murase and Nayar, 1996)

9. Learn and recognise activities Coupled Hidden Markov Models (CHMM) techniques (Oliver, Rosario & Pentland, 1999) Activities with interactions via coupled states in a HMM

10. Learning invariant models Variances for 3 clusters Means for 3 clusters

11. Key Cognitive Elements Visual control and attention – A framework for attentional control – Inferring likely behaviour using Bayes nets – Deictic markers – Attentional selection of objects

12. A Framework For Task Based Visual Control Scene Interpretation …… CONTROL POLICY (WITH STATE MEMORY) FEATURE COMBINATION d1d1 d2d2 dNdN Image Data Driven Task Based Control

13. BBN Inference of likely vehicle tracks Fixed camera gives direct set of dependencies Image Grid Position BBN has size/orient hidden nodes Leaf nodes ls1/2, lo1/2 observables IGP size orient ls1lo1 lo2 ls2 Gong and Buxton, 1993

14. Deictic Markers in inference of behaviour Left: attention for overtake (overtaken & overtaking vehicle) Right: attention for giveway (stopped & blocker vehicle plus ground-plane conflict zone ) Howarth and Buxton,1996

15. Attentional selection using eye gaze

16. Attentional selection using predicted trajectory data

18. Attentional selection using predicted Space of Interest

19. Summary Cognitive Computer Vision is a multi-disciplinary area of research Here we use statistical inference and learning for robust models Task based attentional control is key to prediction and cognitive systems design Useful reference: “Visual surveillance in a dynamic and uncertain world” Buxton, H and Gong, S, Artificial Intelligence 78, pp 431-459, 1995

20. Next time … Generative models – What are they? – Why are they so important to Cognitive Vision?