How natural scenes might shape neural machinery for computing shape from texture? Qiaochu Li (Blaine) Advisor: Tai Sing Lee

Shape from Texture Problem Frequency gradient Orientation gradient

Slants & Tilts of Planes Corentin Massot, et al. Model of Frequency Analysis in the Visual Cortex and the Shape from Texture Problem. IJCV

Scientific Question Mathematically, slant and tilt can be solved based on estimates of frequency and orientation gradients. But how does the brain do it? Conjecture: Brain learns association between images and 3D structures – so upon seeing an image, the brain can infer the underlying 3D structure. Objective: Study images conditioned on each 3D shape to see if there are characteristic image features associated with each slant and tilt, or varied with 3D shape.

Natural Scenes as a media Mathematic Model 3D Perception Natural Scenes Statistical learning Algebra, frequency analysis Association, probabilistic inference Physical models of image formation

Approach Q: Is it possible for the brain to discover image features such as “spatial frequency gradient” and “orientation gradient” from natural scenes? A: We will fit slant-and-tilt planes to 3D range data, and then analyze the images condition on slant and tilt of the plane.

Methodology

What do we HAVE ? Images with depth information (CMU depth dataset) Optical Image Range Image

What do we WANT ? Discover image features from natural scenes associated with different 3D shape (i.e. slant and tilt). Can we see evidence of “spatial frequency gradient”, and “orientation gradient”?

Approach Stage 1 Processing of 3D range data Partition each image into different regions. Fit slant and tilt planes to patches within each region. Stage 2 Analysis of 2D optical image Retina processing Frequency analysis Principal component analysis

Approach Shape 1 Shape 2 Brain Feature 1 Feature 2 Statistical and Frequency Analysis Stage 1 Stage 2

Stage 1

Partition Normalized-Cut algorithm Segmentation 5 Parts Over-Segmentation 30 Parts Segmentation Omission Jianbo Shi, et al. Normalized Cuts and Image Segmentation. PAMI

Compute 3D Shape Fit range data with plane by regression

Computing Precisely Threshold on sum of squared residual (SSR) Small SSR large SSR

Our Database Natural Scenes Patch Set Optical Patches 3D shape

Stage 2

Focus on TEXTURE Retina Processing KILL LUMINANCE SAVE TEXTURE &

Frequency Analysis Use Fast Fourier Transform Focus on frequency information Neurons in V1 can perform windowed Fourier transform. Expectation Some frequency gradient across space within a patch. The farther away, the higher the frequency.

Principal Component Analysis Principal Component Analysis (PCA) Direction of significant variations of data distribution. Neurons are known to be able to discover principle components of input. Expectation in the PCs Frequency gradient (chirp) Orientation gradient

Results

Average power spectrum (radial frequency) A set of patches Top Part Bottom Part

Principal Components Slant 0, Tilt 0. Slant 75, Tilt 0.

Principal Components Slant 45, Tilt 45.

Principal Components Slant 75, Tilt 90. See frequency gradient

Conclusions Preliminary evidence showing spatial frequency gradient can be discovered from natural scenes. Effect is small due to small patch size. Orientation gradient is not evident, but maybe if you use polar angle … Since spatial frequency gradient is correlated with slant and tilt, it is possible that neurons can learn such association.

Future Direction How the brain can learn this association? Simulation on associative learning: Hebbian Learning V1 Shape Neurons V1 Image Neurons V2 Shape Neurons V2 Image Neurons Natural Scenes Hebbian learning

Future Direction Independent component analysis Extract distribution of independent features specific to each slant and tilt. Better for discovering spatial frequency and orientation gradients based on these distributions. Good models for V1 neurons. Prediction: different spatial distribution of features (independent components) for different slant and tilts.

Q&A

THANKS