Vision in Man and Machine. STATS 19 SEM Talk 2. Alan L. Yuille. UCLA. Dept. Statistics and Psychology.

The Purpose of Vision. “To Know What is Where by Looking”. Aristotle. ( BC). Information Processing: receive a signal by light rays and decode its information. Vision appears deceptively simple, but there is more to Vision than meets the Eye.

Ames Room

Perspective.

Curved Lines?

Brightness of Patterns: Adelson (MIT)

Visual Illusions The perception of brightness of a surface, or the length of a line, depends on context. Not on basic measurements like: the no. of photons that reach the eye or the length of line in the image..

Perception as Inference Helmholtz “Perception as Unconscious Inference”.

Ball in a Box. (D. Kersten)

How Hard is Vision? The Human Brain devotes an enormous amount of resources to vision. (I) Optic nerve is the biggest nerve in the body. (II) Roughly half of the neurons in the cortex are involved in vision (van Essen). If intelligence is proportional to neural activity, then vision requires more intelligence than mathematics or chess.

Vision and the Brain

Half the Cortex does Vision

Vision and Artificial Intelligence The hardness of vision became clearer when the Artificial Intelligence community tried to design computer programs to do vision. ’60s. AI workers thought that vision was “low- level” and easy. Prof. Marvin Minsky (pioneer of AI) asked a student to solve vision as a summer project.

Chess and Face Detection Artificial Intelligence Community preferred Chess to Vision. By the mid-90’s Chess programs could beat the world champion Kasparov. But computers could not find faces in images.

Man and Machine. David Marr ( ) Three Levels of explanation: 1. Computation Level/Information Processing 2. Algorithmic Level 3. Hardware: Neurons versus silicon chips. Claim: Man and Machine are similar at Level 1.

Vision: Decoding Images

Vision as Probabilistic Inference Represent the World by S. Represent the Image by I. Goal: decode I and infer S. Model image formation by likelihood function, generative model, P(I|S) Model our knowledge of the world by a prior P(S).

Bayes Theorem Then Bayes’ Theorem states we show infer the world S from I by P(S|I) = P(I|S)P(S)/P(I). Rev. T. Bayes

Bayes to Infer S from I P(I|S) likelihood function. P(S) prior..

Technically very interdisciplinary But applying Bayes is not straightforward. A beautiful theory is being developed adapting techniques from Computer Science, Engineering, Mathematics, Physics, and Statistics. E.G. Probabilistic Reasoning (Pearl CS), Level Sets (Osher Maths).

Examples Generative Models Visual Inference: (1) Estimating Shape. (2) Segmenting Images. (3) Detecting Faces. (4) Detecting and Reading Text.

Generative Models Learn Generative Models from a few images and then generate new images.

Uses of Generative Models Univ. Oxford

Shape Inference: (Zhu Lab)

Shape and Photometry ( Soatto Lab) – Estimate geometry (shape) and photometry from multiple images. Jin-Soatto-Yezzi

Compare ground truth (Soatto Lab) Jin-Soatto-Yezzi 11/1/02 Estimated shape Alternative algorithm Ground truth

Generated Image: synthesized from novel viewpoint and illumination. Jin-Soatto-Yezzi 11/1/02 Ground Truth: same lighting and viewpoint Compare w. ground truth (Soatto Lab)

Segmentation (Level Sets)

Segmenting Images (Zhu Lab) Characterize the set of image patterns that occur in natural images. Provide mathematical models. P(I|S) and P(S).

Face and Text Detection.

Back to the Brain Top-Level; compare human performance to Ideal Observers. Explain human perceptual biases (visual illusions) as strategies that are “statistical effective”.

Brain Architecture The Bayesian models have interesting analogies to the brain. Generative Models require top-down processing

High-Level Tells Low-Level to Shut Up (Kersten Lab)

High-Level Tells Low-Level to Shut up (Kersten Lab)

Conclusion Vision is unconscious inference. Theory of Vision for Man and Machine. See more about Vision at UCLA in the Vision and Image Science Collective