1. Manifold Bootstrapping for SVBRDF Capture
Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Moshe Ben-Ezra, Yanxiang Lan, Baining Guo
Tsinghua University Microsoft Research Asia Microsoft Research
Good afternoon, what we do is a new method to capture high quality surface reflectance in just a few minutes and with a few capturing data.

2. High-Quality SVBRDF Acquisition
high spatial variation
high angular variation
Real-world materials exhibit rich appearance with detailed variation in both spatial and angular.
A high-quality SVBRDF that captures all these details is critical for realistic image synthesis.
fast and simple too!

3. Related Work brute force (6D) measurement (gonioreflectometer) slow
[Dana et al. 1999, McAllister et al. 2002, Lawrence et al. 2006]
slow
expensive, specialized rig
The straightforward approach is to directly sample the six D reflectance function.
This is very time-consuming because the data needed is very high-dimensional.
Also it requires dedicate devices that are expensive.

4. Related Work single-pass fitting methods measures large dataset
[Lensch et al. 2003, Goldman et al. 2005,
Zickler et al. 2005]
measures large dataset
fits limited models (parametric/isotropic)
Single-pass data-driven approach exploit redundancy in reflectance function to reduce capture time and data. While, reducing the number of measurements scarifies either spatial or angular resolution.
They are also limited to parametric BRDF models or isotropic BRDFs, which do not fit all real-world materials accurately.

5. Related Work two-pass methods
linearly combine two representatives based on diffuse color [Debevec et al. 2003]
Recent methods separate data acquisition into two passes, one for angular and one for spatial variation.

6. Related Work two-pass methods
linearly combine two representatives based on diffuse color [Debevec et al. 2003]
Debevec et al measure a set of representative BRDFs from the scene, and represent BRDFs by linear combination of two representatives base on similarity of their diffuse colors.
This approach works well for acquiring nearly diffuse surface, but fails in general for complex and specular materials.

7. Related Work two-pass methods
linearly combine two representatives based on diffuse color [Debevec et al. 2003]
use existing BRDF database of representatives: non-specialized and isotropic [Matusik et al. 2003b; Weyrich 2006]
Matusik et al. and Weyrich o-six
represent an isotropic BRDF as a linear combination of BRDFs chosen from an existing BRDF database, which is not specialized to the target material and also limited to a small number of isotropic BRDFs.

8. Observation BRDF spatial variation is complex:
tangent/normal/local frame rotates
specularity/anisotropy varies
specular lobe’s falloff and cross-section changes
forms low-dimensional manifold over given target.
manifold isn’t globally linear [Matusik et al. 2003a]
manifold is locally linear.
spatial variation of BRDFs is complex. Sources of its variation include specularity, anisotropy, local frame rotation and the shape of the specular lobe.
Despite its complexity, the spatial-varying BRDFs over a given material still forms a low-dimensional manifold, which isn’t globally linear, but *is* linear when considered at a sufficient small subspace.

9. SVBRDF Manifold locally linear globally non-linear
This example illustrates the SVBRDF manifold. Each disk visualize a four D BRDF by its top-view slice.
[ click ] it is nonlinear in global scale and
[ click ] is linear in local scale.

10. Local vs. Global Interpolation
local interpolation
Local linear interpolation among these three neighborhood BRDFs gives a plausible result.

11. Local vs. Global Interpolation
While global linear interpolation produces implausible ghosting results because it ignores the manifold structure.

12. SVBRDF Manifold Bootstrapping
Representative
Space
To summarize our approach, this diagram represents an SVBRDF manifold.
[Click]
We approximate its overall structure based on hundreds of representative BRDFs. We call it, representative space.

13. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
These representatives are BRDFs with high-resolution details over the angular domain

14. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
But are sparsely sampled over the spatial domain.
Now we need to consider how to reconstruct BRDFs for each pixel over the spatial domain.
Material Sample

15. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
We capture images of the entire surface under several different lighting conditions, which provides a key for each pixel that characterize its BRDF.
Key Measurements
Material Sample

16. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
every pixel
These keys from target surface form the key space, which densely sampled over the spatial domain while is sparse in the angular domain.
We expected, the key space will have the same manifold structure of the representative BRDF space when the number of lighting conditions are sufficient.
Key Space
Key Measurements

17. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space x
Now, we consider reconstruction of the high resolution BRDF at location X.
[Click]
First we find its local linear embedding in the key space. This is represented as a set of neighbors and interpolation weights.
Local Embedding In Key Space
Key Space
Key Measurements

18. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space x
And then we apply the same local linear embedding to the corresponding neighborhood in the representative space.
Local Embedding In Key Space
Key Measurements

19. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
Local Embedding of x In Representative Space x
In this step, low dimensional data is used to generate high dimensional data, which we called bootstrapping.
Local Embedding In Key Space
Key Measurements

20. SVBRDF Manifold Bootstrapping
Representative
Measurements
Representative
Space
Reconstructed
BRDF of x
Local Embedding of x In Representative Space x
Finally, we reconstruct a high resolution BRDF for the location x by linear interpolating neighborhood BRDFs.
The same reconstruction is preformed at each pixel to obtain the full SVBRDF data.
Local Embedding In Key Space
Key Measurements

21. Results Real Material Sample
Here we show a reconstructed wrapping paper, which is based on tens of photos and tens of representative BRDFs.
Notice the sharp pattern, and the glossy highlights are well preserved.
Real Material Sample

22. Outline Data Acquisition SVBRDF Reconstruction Validation
Now we dive into more details of the SVBRDF bootstrapping.
[ click ]
First, data acquisition

23. Representative BRDFs portable BRDF scanner
6 LED light directions, 320x240 view directions
data amplification by microfacet model
0.1s per BRDF
In our experiments, we built a portable BRDF scanner, which captures high resolution microfacet BRDFs at ten per second.
Note that, the proposed manifold bootstrapping approach is not limited to microfacet BRDFs.
Other measurement devices can also be used for capturing representative BRDFs.

24. Key Measurements fixed camera background environmental lighting
+ moving area source
The key measurement is based on the background environmental lighting plus a moving area source.
A light probe located near the material sample captures the lighting applied.
Both the sample and the light probe are captured with a fixed camera.

25. Timing representative BRDFs and key measurements data processing
10-15 minutes
data processing
less than 5 minutes
Our method is fast: besides the device setup, it takes a few minutes for capturing representative BRDFs and another few minutes for the key measurements.
The followed data processing such as HDR construction and spherical re-sampling takes less than five minutes.

26. Outline Data Acquisition SVBRDF Reconstruction Validation
SVBRDF is then reconstructed based on these captured data.

27. SVBRDF Reconstruction
Representative BRDFs
With the captured representative BRDFs, [ click ]

28. Representative Local Interpolation
BRDF of x ?
= w1
+ w2
+ w3 x
Material Sample
Representative BRDFs
The SVBRDF is reconstructed by solving BRDF pixel by pixel.
[ click ]
For each pixel, its BRDF is regarded as a local linear interpolation in the representative BRDF space.

29. Representative Local Interpolation
choose which representatives to interpolate from
solve for weights wi
BRDF of x ?
= w1 w1
+ w2 w2
+ w3 w3 x
Material Sample
Representative BRDFs
Which means we need to determine which representative BRDFs to interpolate from.
[Click]
And, solve the interpolation weights.

30. Projected Keys of Representative BRDFs
Key Measurement
Environment Lighting
Representative BRDFs
Material Sample
We based on the key measurements at each pixel to determine which representative BRDFs to interpolate from and how.
[Click]
We also obtain projected key measurements of representative BRDFs by rendering with the same environment lighting.
Key Measurements
Projected Keys of Representative BRDFs

31. Projected Keys of Representative BRDFs
Key Measurement
Key Measurements
Projected Keys of Representative BRDFs
We determine the local linear embedding based on the relationship between the key measurement at each pixel and that of representative BRDFs.

32. Key Local Interpolation
Projected Keys of Representative BRDFs
nearest neighbor in key space
Key Measurements x
Key of x
With the key measurement at pixel X,
[click]
We find the k-nearest neighbors in the set of key measurements of representatives BRDFs.

33. Key Local Interpolation
solve for weights: LLE [Roweis & Saul 2000]
where
Key Measurements x
Key of x
= w1
+ w2
+ w3
We solve the best linear reconstruction with these neighborhood keys measurement of representatives.
[Click]
It determines the weights based on the distances in key space.

34. BRDF Reconstruction Neighborhood = w1 + w2 + w3 Key of x
Finally, We use the corresponding representative BRDFs of the neighborhood in key space,
Local Embedding
in Key Space

35. BRDF Reconstruction = w1 + w2 + w3 = w1 + w2 + w3 weights BRDF of x
Key of x
= w1
+ w2
+ w3
[Click]
As well as the weights, to generate the high resolution BRDF at location x.
Local Embedding
in Key Space

36. Outline Data Acquisition SVBRDF Reconstruction Validation
The proposed SVBRDF bootstrapping assumes the key space and the representative space have the similar structure. Is that always the case?

37. Key Space vs. Representative Space
Projection depend on the environmental lighting conditions
preserve distances ⇒ preserve BRDF manifold structure
We evaluate the similarity in manifold structure based on the distance of corresponding points in the two spaces.
It is to say, distances of representative BRDFs should be preserved in the projected key space.

38. Key Space vs. Representative Space
Projection depend on the environmental lighting conditions
preserve distances ⇒ preserve BRDF manifold structure
global distances
⇒ preserve neighborhoods
local distances
⇒ preserve weights
Preserving distance in global scale ensures that neighbors in key space remain neighbors in BRDF space.
[Click]
Preserving distance in local scale ensures that the interpolation weights are consistent in the two spaces.

39. Distance Preservation
preservation evaluation
To evaluate distance preservation, we compare the distances of representatives in the projected key space and the BRDF space.
If the ratio is close to 1, the key space is a good proxy for the BRDF space, and it will be safe to apply local linear embedding in key space to BRDF space.

40. Distance Preservation
preservation evaluation
# of lighting conditions
The major factor that affects distance preservation is the number of lighting conditions. Which also determines the dimension of the projected key space.

41. Distance Preservation
preservation evaluation
# of lighting conditions
criterion: global: τg > local: τl > 0.85
A typical case is shown here. With the number of lighting condition increasing, the distance preservation rise quickly and approaching one with tens of lighting conditions.
[ click ]
In our experiments, we ensure the distance preservation is greater than O point nine in global scale and O point eight five in local scale.

42. Results Real Material Sample
Here we show a result with SVBRDF captured by the proposed method.
Real Material Sample

43. Extension to local frame variations
Normal variations
Tangent rotations
We also extend the proposed method to account for local frame variations so that we can handle materials that has normal variation and tangent rotations.

44. Representative Enlargement…
enlarged BRDFs over normal rotation …
enlarged BRDFs over tangent rotation
After the representative BRDFs are captured, we synthetically enlarge the set of representatives by rotating the BRDF data for possible normal orientations and tangent rotations.
Then we apply the same bootstrapping method with the extended representative BRDFs which includes all expected local frame variations.

45. Results Real Material Sample
Here is a rendering result of a measured SVBRDF with bumpy surface.
Real Material Sample

46. Results Real Material Sample
Now we show a highly anisotropic results with tangent rotations.
Notice the fine detail of the brushing is well reproduced.
Real Material Sample

47. Conclusion Manifold bootstrapping captures high-resolution SVBRDF
assumes BRDF forms low-dimensional manifold
decomposes acquisition into two phases
makes sparse measurement in both
phase one (representatives) = sparse spatial, dense angular
phase two (keys) = sparse angular, dense spatial
simplifies and accelerates the capture process
In conclusion, the proposed manifold bootstrapping acquires an SVBRDF with high resolution in both the spatial and angular domains.

48. Conclusion Manifold bootstrapping captures high-resolution SVBRDF
assumes BRDF forms low-dimensional manifold
decomposes acquisition into two phases
makes sparse measurement in both
phase one (representatives) = sparse spatial, dense angular
phase two (keys) = sparse angular, dense spatial
simplifies and accelerates the capture process
It assumes reflectance variation over the target surface is a low-dimensional manifold.
By decomposing acquisition into two phases, and making only sparse measurements in each, it simplifies and accelerates the capture process.

49. Acknowledgements Paul Debevec for HDR images
Steve Lin for video narration
Anonymous reviewers for helpful comments
We thanks many peoples for their help in our paper.

50. Thanks
Thanks for your attention.

51. Thanks
EOP !!

52. Uniform Measurement Scaling
Representative Projection

53. Uniform Measurement Scaling
Representative Projection

54. Future Work improving the hand-held BRDF scanner
handling self-shadowing and masking effects
For the future work, the hand-held BRDF measurement device can be further improved by avoiding optical artifacts, and extended to non microfacet model BRDFs.
Our method does not handle materials that exhibit significant self-shadowing and masking effects, so we’d like to extend it to that class of materials.

55. Implementation capturing parameters: Material Sample Key Measurement
Representative BRDFs
resolution
number /
glossy paper
1000x1000 50 30
0.90 / 0.87
wrinkled paper
1000x600
200
30*3600
0.90/0.83
aluminum pan
2000x2000
10*360
0.99/0.85
This slide summarizes parameters used in capturing.
The spatial resolution acquired is high and captures very small spatial features.
For an isotropic material, 50 key measurements is sufficient. [JMS: what is this material? Is it isotropic or anisotropic? Why is 50 measurements sufficient? Is there a simple rule of thumb that tells how many key measurements are necessary for various kinds of materials?]
Even with complex normal or tangent variations, only 200 key measurements can faithfully capture the material appearance details.
[JMS: show small images of each material.]
[JMS: add “old copper”?]
[JMS: It’s really too bad we didn’t have more examples by Siggraph presentation time. Such a small number isn’t very convincing.]