1. Mingjing Zhang and Mark S. Drew
ECCV-CPCV2012
Robust Luminance and Chromaticity for Matte Regression in Polynomial Texture Mapping
Mingjing Zhang and Mark S. Drew
School of Computing Science, Simon Fraser University, Vancouver, Canada

2. Overview Introduction PTM Model (Polynomial Texture Mapping)
Brief recapitulation of robust approach:
Outlier Identification
- Surface properties: Colour, Albedo and Normal
Specularity and Shadow Interpolation
- light direction interpolation
Reduction in complexity – luminance/chrominance outliers, dimensional reduction

3. What’s the idea?
PTM uses multiple input images, for fixed camera but with changing lighting direction. So, like Photometric Stereo (PS).
But uses many lights (50, say) == polynomial in light-direction vector components.
So, idea → know light directions
→ change light, observe changing shading
→ obtain Colour, Albedo, and Normal
This idea well known in painting:

5. Typical dome
Worcester Art Museum, Boston, images © Cultural Heritage Imaging

6. Standard PTM is based on a poly
Standard PTM is based on a poly. regression of light-direction 3-vector to RGB;
but doesn’t actually interpolate specularities (highlights) or shadows!

7. Interpolation of light angles
The Aim
First we would like to detect specularities and shadows in the set of input images.
Known lighting angle 1
Known lighting angle 2
Interpolation of light angles
…And then we would like to interpolate to re-light the object under a new, non-measured lighting direction.

8. and, what’s the shadow problem?
For example, assume a pole with two light directions, as depicted in Figure 1. The shadows are cast in different directions. If we apply a naïve image-based interpolation we get a cross-dissolve, which is clearly wrong (Figure 2).
(Example due to Yacov Hel-Or)
Figure 1: Two images taken of a pole standing on a plane.

9. …and, what’s the problem…
Figure 2: Left — desired result. Right — image-based interpolation.

10. Robust Methodology [Image and Vision Computing, vol. 30, 2012, pp
Solve the PTM model in a robust version  leads to identification of inliers and outliers.
Generate surface normals, surface albedo, and chromaticity, using inliers.
Model specularity and shadow using RBF regression over outliers found.

11. N.B. 2 objectives: accurate surface properties
convincing/revealing appearance

12. Again: What is PTM?
n images of a scene from n different lighting directions:
single camera, single viewpoint, no motion, multiple lights.
The National Gallery, London

13. PTM Model
PTM : generalization of PS → non-linear polynomial regression.
Poly. can better model intricate dependencies due to self-shadowing.
The aim is to find vector c, =regression coefficients, at each pixel position — Suppose we have n lights from normalized directions a i , i = 1..n :
Matte surface!
n x 6
6 x 1
n x 1

14. Modified PTM Model
We use robust regression (uh-oh) to find the coefficients.
Suppose we happen to have a Lambertian surface;
then get normal n and albedo α exactly:
Note:
Suppose we indeed happen to have a Lambertian surface; since
we mean to use a robust regression to solve for c , then regardless of specularities or shadows
(assuming at least half plus one of the pixel values at an image location are non-outliers), the
regression will just generate the correct surface normal vector, multiplied by a surface albedo
times a lighting strength factor. The regression will place zeros in c for the higher order
terms. Nevertheless, it is useful to keep a polynomial description, to suit surfaces which are
not Lambertian. Note that if we were to use instead p = fu;v;wg the method would simply
reduce to PST. Here we regress making no assumption about a Lambertian character of the
surface, and can reconstruct pixel values without making any such assumption
If at a pixel the collection of luminances are Lambertian+
shadow + spec., using robust approx’ly still get correct
matte regression coefficients.

15. Non-Robust PTM and Robust-PTM on a Synthetic Sphere
too bright
Specularity
Shadow
Synthetic sphere
Regenerated matte sphere using non-robust PTM
Regenerated matte sphere using robust PTM
Matte

16. Compare Robust to LS: LS Matte → Robust Matte →
Worcester Art Museum, Boston
LS Matte →
leftover spec’s
Robust Matte →

17. Highlight and Shadow Identification…
Single pixel, 50 lights (matrix A, 50x3): generate tri-partite
weights +1=specular, 0=matte, -1=shadow

18. Surface Normal and Surface Albedo
Solid lines: Noise sensitivity for robust PTM estimate of surface normal and
albedo. Dot-dashed lines: robust PST – using weights w0 generated by robust PTM. Dashed
lines: standard PST. (b): Synthetic sphere with 10% noise. (c): Noise contribution (positive
values shown).
In comparison, standard PST, based on straightforward least squares including all pixel
values, has poorer estimates – dashed lines in Fig. 4 – because the effect of noise is swamped
by the main problem, inclusion of shadow and specular values.
Fig. 4 shows results over increasing percent Gaussian noise, for a synthetic sphere RGB
image, with pixel values generated for 50 lighting directions for a Lambertian surface, plus
Phong illumination [12] with roughness 1/20, plus noise. Here the solid lines show the results
for error in normal vector direction (blue) and albedo (red), from eq. (8). Unsurprisingly, if
there is indeed no noise, and the base reflectance is Lambertian, a robust regression ignores
the polynomial terms in the regression model and
This is because the extra polynomial terms in PTM will tend to
over-fit the correct underlying noise-free values, whereas (9) assumes a Lambertian model,
correctly in this synthetic case.
PST
Robust PST
PTM Coefficients

19. Some real data:

20. Chromaticity We know inliers (matte values) at each pixel position.
The chromaticity is RGB triple divided by Luminance
A good estimate for chromaticity is:

21. Modeling Specularities and Shadows
To model the dependency of specularity and shadow on lighting direction, we use RBF:
not explained by Matte
The reason for including all non-sheen
pixels in the shade s , and not just w-, is that then the two RBF interpolations, plus
matte, combine to exactly equal the input luminance images.
RBF Coefficients
“Polynomial” term
Gaussian RBF

22. Results normals closest in-sample some interpolants
In situations where shadows are completely black, as in Fig. 3(b), the method can produce
streaks resulting from hard shadow regions. Nevertheless even for cases where pixels
are saturated, as in Fig. 3(d,f), the sheen model simply sees these as extra, bright information
and successfully models them in interpolants.

23. Harder Interpolation: Shiny Material!
Actual and this method
Actual and this method
Interpolated 
Original poor representation
Original very poor
interpolation 
Original poor representation

24. Summary
Interpolate specularity and shadows within the PTM framework
== Obtain accurate surface properies
== Obtain an accurate RGB rendering under new lighting
Next examine mechanisms for reducing the space and time complexity of the robust matte modeling:

25. < These attacks on this problem are a work in progress! >
Recall: for Matte we use robust regression (uh-oh) to find the coefficients.
Time-complexity of robust is O(n6 log n) with n50: slow!
Could keep robust, but reduce from 6-D to 1-D: now O(n log n) → much better!
< These attacks on this problem are a work in progress! >
Or, strike a balance by going to 3-D to 1-D → ok.

26. Could keep robust, but reduce from 6-D to 1-D: now O(n log n) – much better:
This is “mode-finding” … apply to Luminance or to each component of Chrominance 2-vector: this still gives robust outlier identification (the slow part!)
Then apply usual 6-D to 1-D, but on inliers == trimmed LS.

27. 6-D →Lum.
1-D Chrom. (green)
1-D Lum.

28. Gaussian-noise sensitivity for synthetic data:
best 6-D
mode 1-D
 linear 3-D

29. What we learn here is that a robust approach does pay, but we can use a simpler (albeit still robust) method, at least for such basically Lambertian plus shadows plus specularity data.

30. (a,b): Input images with lighting directions to be interpolated; (c): interpolants
for direction given by the mean of directions for (a,b); (d): robust matte image for
interpolated angle; (e): normals.

31. to the Natural Sciences and Engineering Research Council of Canada
Thanks!
to the Natural Sciences and
Engineering Research Council of Canada
and Hewlett-Packard Labs