Jochen Triesch, UC San Diego, 1 COGS Visual Modeling Jochen Triesch & Martin Sereno Dept. of Cognitive Science UC San Diego

Jochen Triesch, UC San Diego, 2 What is this course about? it’s about vision: the ability of people, animals, and machines to see things it’s interdisciplinary — it brings together computer vision, visual neuroscience, visual psychophysics it’s team-taught it’s “hands on” : exercises, project, and paper! it requires math (calculus, linear algebra, statistics, complex numbers) it requires programming (Matlab) it teaches you what you need to know to do state-of- the-art research in computational vision

Jochen Triesch, UC San Diego, 3 Contents fundamentals: light and imaging, overview of anatomy and physiology of primate visual system low-level vision: image processing (convolution, morphological operators, pyramids), edges, corners, color, stereo vision, shading, texture, local motion mid-level vision: segmentation, grouping, contour processing, cue integration high-level vision: object recognition, attention, active vision, visual memory, vision and other modalities How do people learn to do it? How can we get computers to learn to do it?

Jochen Triesch, UC San Diego, 4 Textbook fundamentals low-level vision mid-level vision high-level vision: recent research papers lecture notes will be posted on the course web page (linked from my home page)

Jochen Triesch, UC San Diego, 5 Vision is really, really hard! forward problem (graphics) : from world to images has unique solution inverse problem (vision): from images to description of the world does not have unique solution: infinitely many scenes can give rise to the same image

Jochen Triesch, UC San Diego, 6 The need for studying computational vision David Marr: “Trying to understand perception by studying only neurons is like trying to understand bird flight by studying only feathers: it just cannot be done. In order to understand bird flight, we have to understand aerodynamics; only then do the structure of the feathers and the different shapes of birds’ wings make sense.”

Jochen Triesch, UC San Diego, 7 Marr’s levels of analysis (1982) Computational theory: What is the goal of the computation, why is it appropriate, and what is the logic of the strategy by which it can be carried out? Representation and algorithm: How can this computational theory be implemented? In particular, what is the representation for the input and output, and what is the algorithm for the transformation? Hardware implementation: How can the representation and algorithm be realized physically?

Jochen Triesch, UC San Diego, 8 Why have brain? To control body. Why have perception? To acquire information about what way of moving your body may be particularly useful now. Thus, what you perceive affects what you’re going to do. But also, what you do changes what you perceive. OrganismEnvironment Action Perception This feedback loop can result in surprisingly complex looking behavior patterns! Perception and Action

Jochen Triesch, UC San Diego, 9 Some Current Challenges I vision is for action; how do you (learn to) extract the information that you need for accomplishing things? how is vision integrated with action, i.e. how are perception and behavior orchestrated? E.g., where should you look next and why? What parts of the scene should you attend to or ignore, what pieces of information should you extract at any time? in how far is visual processing reactive or goal-driven?

Jochen Triesch, UC San Diego, 10 The Vision Problem Object Tracking Structure from Motion Stereo Vision Segmentation Object Recognition Color Vision Shape from Shading The Vision Problem Object Tracking Structure from Motion Stereo Vision Segmentation Object Recognition Color Vision Shape from Shading Some Current Challenges II understand vision as a complex problem

Jochen Triesch, UC San Diego, 11 Some Current Challenges III where does the solution come from? nature and nurture can we build a “sentient statue”? (Étienne Bonnot de Condillac) what does it take to learn to see without supervision?