Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of Jerusalem Joint work with Assaf Zomet and Yair Weiss

Transparency

? How does our visual system choose the right decomposition ? Why not “simpler” one layer solution? Which two layers out of infinite possibilities?

Talk Outlines Motivation and previous work Our approach Results and future work

Transparency in the real world “Fashion Planet's photographers have spent the last five years working to bring you clean photographs of the windows on New York especially without the reflections that usually occur in such photography”

Transparency and shading

Transparency in human vision Metelli's conditions (Metelli 74) T-junctions, X-junctions, doubly reversing junctions (Adelson and Anandan 90, Anderson 99) Two layersOne layer Not obvious how to apply “junction catalogs” to real images.

Transparency from multiple frames Two frames with polarizer using ICA (Farid and Adelson 99, Zibulevsky 02) Multiple frames with specific motions (Irani et al. 94, Szeliski et al. 00, Weiss 01)

Shading from a single frame Retinex (Land and McCann 71). Color (Drew, Finlayson Hordley 02) Learning approach (Tappen, Freeman Adelson 02)

Talk Outlines Motivation and previous work Our approach Results and future work

Our Approach Ill-posed problem. Assume probability distribution Pr(I 1 ), Pr(I 2 ) and search for most probable solution. (ICA with a single microphone)

Statistics of natural scenes Input imagedx histogramdx Log histogram

Statistics of derivative filters Log histogram Generalized Gaussian distribution (Mallat 89, Simoncelli 95) Gaussian –x 2 –x 1/2 0 Log Probability Laplacian –|x|

Is sparsity enough? =+=+ Or:

Exactly the same derivatives exist in the single layer solution as in the two layers solution. Is sparsity enough? =+ =+ Or:

Beyond sparseness Higher order statistics of filter outputs (e.g. Portilla and Simoncelli 2000). Marginals of more complicated feature detectors (e.g. Zhu and Mumford 97, Della Pietra Della Pietra and Lafferty 96).

Corners and transparency In typical images, edges are sparse. Adding typical images is expected to increase the number of corners. Not true for white noise =+

Harris-like operator

Derivative FilterCorner Operator Corner histograms

Fitting: Derivative FilterCorner Operator Typical exponents for natural images:

Simple prior for transparency prediction The probability of a decomposition

Does this predict transparency?

How important are the statistics? Is it important that the statistics are non Gaussian? Would any cost that penalized high gradients and corners work?

The importance of being non Gaussian

The “scalar transparency” problem Consider a prior over positive scalars For which priors is the MAP solution sparse?

The “scalar transparency” problem Observation: The MAP solution is obtained with a=0, b=1 or a=1, b=0 if and only if f(x)=log P(x) is convex MAP solution: a=0, b=1MAP solution: a=0.5, b=0.5

The importance of being non Gaussian

? Can we perform a global optimization ?

Conversion to discrete MRF Local Potential- derivative filters: Pairwise Potential- pairwise approximation to the corner operator: -Enforcing integrability For the decomposition: gradient at location i

Conversion to discrete MRF Local Potential- derivative filters: Pairwise Potential- pairwise approximation to the corner operator: -Integrability enforcing For the decomposition:

possible assignments. Solution: use max-product belief propagation. The MRF has many cycles but BP works in similar problems (Freeman and Pasztor 99, Frey et al Sun et al 2002). Converges to strong local minimum (Weiss and Freeman 2001) Optimizing discrete MRF

Drawbacks of BP for this problem Large memory and time complexity. Convergence depends on update order. Discretization artifacts

Talk Outlines Motivation and previous work Our approach Results and future work

Results inputOutput layer 1 Output layer 2

Results inputOutput layer 1 Output layer 2

Future Work OriginalNon linear filter Dealing with a more complex texture +=

Future Work Dealing with a more complex texture: Use application specific priors (e.g. Manhattan World) Extend to shading and illumination. Applying other optimization methods. Learn discriminative features automatically A coarse qualitative separation.

Conclusions Natural scene statistics predict perception of transparency. First algorithm that can decompose a single image into the sum of two images.