Specularity, the Zeta-image, and Information-Theoretic Illuminant Estimation Mark S. Drew1, Hamid Reza Vaezi Joze1, and Graham D. Finlayson2 2School of Computing Sciences, The University of East Anglia Norwich, England graham@cmp.uea.ac.uk 1School of Computing Science Simon Fraser University Vancouver, BC, Canada mark@cs.sfu.ca

Zeta-Image: Goal= Discover the light color (yet another!??) Relative Chromaticity: The main idea is that we can get at a good solution for the chromaticity of the light by dividing image chromaticity 3-vector  by the candidate light chromaticity e — the “relative chromaticity”  =    e [where   is component-wise division].

Algorithm: Then we show that, over pixels that are specular or white, the log of the relative chromaticity log() is perpendicular to the light chromaticity e in color space. This gives a useful hint for recovering e .

Proof; Background: Simple image formation model -- k = R,G,B surface “Neutral interface”: Lee, JOSA 1986, “Method for computing the scene-illuminant hromaticity” k = R,G,B pixel color surface light specularity Light is “white enough”: Borges, JOSA 1991, “Trichromatic approximation method for surface illumination”

Proof…  = {R,G,B}/{R+G+B) Now chromaticity: Relative chrom.:

Proof… …Relative chrom.: simple!

Proof… Now let’s head for a Planar Constraint:

Proof…

Therefore, let the Zeta-image† be Definition: where e is the chromaticity of illuminant and (x) is the chromaticity value of pixel at position x . Properties It has the structure of the Kullback-Leibler Divergence from Information Theory; Zeta is low (near-zero) at specularities: † Patent applied for

Zeta-image for illumination estimation Planar constraint: For near-specular pixels or white surfaces, Log-Relative-Chromaticity values are orthogonal to the light chromaticity. Or, equally, the zeta image is near zero. So the best light chromaticity e for an image is that which minimizes the zeta-image for near-specular pixels (or white surfaces) => guess a domain  ; then , == Search

Search — explained: assume candidate e; form  ; the lowest 10-percentile, say, of dot-product values could be near-specular pixels. Over a grid of possible light-chromaticities e, minimize dot-product values over candidate illuminants for those lowest 10- percentile pixels.

Or, Thm: Analytic solution is the geometric mean of where  is a set of bright pixels Search is better than Analytic but slower.

Domain  for Analytic Solution: We start by approximating  as top-5% brightness pixels. Could be any other method to indicate near- specular pixels and white surface regions. Detecting Failure  i.e., detecting images not having specularity or white surfaces in top brightness pixels:  can stem from areas of images belonging to the brightest surface which happens to tend to be some particular surface color. we can simply check if these pixels are in the possible chromaticity gamut of illuminants.  can be a bag-of-pixels from all over the image. we can investigate the distribution of  in chromaticity space.

Does this work?... divide by correct light, bottom 20% of Zeta: yes!

...Does this work?... incorrect light, bottom 20% of Zeta: no! correct light, float (inverted) Zeta 

Details of Analytic: How did we come to the geometric mean? Solve:  

…Details of Analytic… How do we know  is positive? If components were probabilities then has structure of Kullback-Leibler Divergence: extra bits to code samples from ek when using codebook based on ik , so positive!

…Details of Analytic… Does this work? Final step: form  using the geomean chromaticity for bright pixels, and then trim to least-10% values of Zeta, and recalculate the geomean. Does this work?

float (inverted) Zeta bright lowzeta = bright & (zeta<quantile(zeta)),0.10);

plots of  for correct light e (O) and for analytic solution e (x) :

In expts. mask off the ColorChecker

Algorithm 2: Planar Constraint Applied as Post-Processing so far: Algorithm 1: Use either analytic answer, or a simple hierarchical grid search over light- chromaticity Algorithm 2: Planar Constraint Applied as Post-Processing Use any alg’s answer for e; take SVD of lowest-10% dot-product pixels: improves light estimate!

fallback position: If either of two chromaticity checks fails, use Grey-Edge.

Post-processing: Results.

Illumination Estimation: Results. (Analytic)

Funding: Natural Sciences and Engineering Research Council of Canada Thanks! Funding: Natural Sciences and Engineering Research Council of Canada