1. 1 Compression and Real-time Rendering of Measured BTFs using local-PCA Mueller, Meseth, Klein Bonn University Computer Graphics Group

2. VMV 2003  University of Bonn  Computer Graphics Group  2/25 2 MotivationMotivation  (Real-time) Rendering of complex meso-structure:  Shadowing  Masking  Light-Transport:  inter-reflections  sub-surface scattering  etc.  Classical modeling and rendering approach infeasible

3. VMV 2003  University of Bonn  Computer Graphics Group  3/25 3 MotivationMotivation  Common „work around“: Meso-structure is rendered from one image Meso-structure is rendered from one image  Texture mapping + Fast and simple + Hardware support – Flat appearance – Only simple relighting

4. VMV 2003  University of Bonn  Computer Graphics Group  4/25 4  Improvement: Meso-structure is rendered from many images Meso-structure is rendered from many images  Rendering measured BTF Bidirectional-Texture-Function (Dana et al. 1999) Bidirectional-Texture-Function (Dana et al. 1999) Light and view-dependent Texture Light and view-dependent Texture Apparent BRDF that varies per texel Apparent BRDF that varies per texel + Captures all light and view- dependent effects of a material MotivationMotivation

5. VMV 2003  University of Bonn  Computer Graphics Group  5/25 5 ProblemProblem  Accurate samplings of the 6-dimensional BTF( x, y,  i  r ) contain many images  e.g. Sattler et al.(Bonn University, 2003):  81 directions for light and view each  256x256 texel spatial extend  6561 RGB-images ~1.2GB Interpolation from sampled data impossible in real- time Interpolation from sampled data impossible in real- time  Memory reduction required http://btf.cs.uni-bonn.de/

6. VMV 2003  University of Bonn  Computer Graphics Group  6/25 6 Previous Work  McAllister et al. (2002)  Fitting analytical BRDF-model (generalized cosine lobes - Lafortune) to every texel + Fast rendering and high compression (~1:500) – Limited quality (depth impression!) – Non-linear fitting required (expensive, <5 lobes) McAllister 2002

7. VMV 2003  University of Bonn  Computer Graphics Group  7/25 7 Previous Work  Daubert et al. (2001)  Fitting Lafortune to synthetic cloth-BTFs  Including view-dependent scaling factor + Increased depth-impression and moderate memory requirements (~1:40) – Modeling abilities still limited: white plaster per-texel apparent BRDF Lafortune 2 lobes iiii rrrr scale factor 2 lobes Lafortune 10 lobes scale factor 10 lobes Daubert et al. 2001

8. VMV 2003  University of Bonn  Computer Graphics Group  8/25 8  Meseth et al. (2003)  Fitting of analytical functions (polynomials, lobes) for fixed measured view-direction (reflectance fields)  Rendering employs view-interpolation + Masking and shadowing captured – High memory requirements (~1:15) – Artificiality of the fitted analytical functions still notable Previous Work Meseth et al. 2003 iiii rrrr originalper-view polynomial

9. VMV 2003  University of Bonn  Computer Graphics Group  9/25 9 Previous Work  Suykens et al. (2003)  Application of an improved BRDF-factorization technique (Chained-Matrix-Factorization, CMF)  Clustering of resulting factors leads to compact representation + Fast implementation on current graphics hardware – Tested samples not representative for real measured BTFs Suykens et al. 2003 iiii rrrr CMF (four factors) no clustering original

10. VMV 2003  University of Bonn  Computer Graphics Group  10/25 10 Previous Work  Sattler et al. (2003)  Perform PCA on images with fixed view direction  Combine the resulting “Eigen-Textures” during rendering + High quality + Environmental lighting supported – High memory requirements (~1:10) – Real-time only for small meshes (CPU-operations per vertex) Sattler et al. 2003 iiii rrrr original 8 PCA components

11. VMV 2003  University of Bonn  Computer Graphics Group  11/25 11 Our Approach  Interpret the measured BTF as set of high- dimensional vectors (either images or per texel apparent BRDFs)  Expect correlation between vectors  Apply data analysis tools for dimensionality reduction reprojected imagesper-texel apparent BRDF iiii rrrr

12. VMV 2003  University of Bonn  Computer Graphics Group  12/25 12 Our Approach  Generalize Sattler et al.:  Cluster data to subsets  Apply Principal Component Analysis (PCA) to data in that subsets  Piece-wise affine-linear approximation: affine-linear approximation3 piece affine-linear approximation

13. VMV 2003  University of Bonn  Computer Graphics Group  13/25 13 Our Approach PCA basis vector  How should we cluster?  Generalized Lloyd-algorithm  Euclidean distance  Reconstruction error Local-PCA (Kambhatla, Leen [1997])

14. VMV 2003  University of Bonn  Computer Graphics Group  14/25 14 Analysis Average reconstruction error (proposte, c=8) original reconstruction (k=32, c=8) imagesBRDFs inverted difference  BRDF-arrangement performs superior  Represent BTF by sets of “Eigen-BRDF”s cluster index map c=30,k=1 no clustering

15. VMV 2003  University of Bonn  Computer Graphics Group  15/25 15 Analysis - Comparison 1.2GB 32 MB 32 MB 106 MB 60 MB 60 MB 121 MB 10 MB per-view PCA (c=8) CMF (four factors) no clustering per-view polynomial scale factor 2 lobes LPCA (k=32, c=8) iiii rrrr original white plaster iiii rrrr corduroy

16. VMV 2003  University of Bonn  Computer Graphics Group  16/25 16 Analysis - Results raw datacompressed (k=32, c=8)

17. VMV 2003  University of Bonn  Computer Graphics Group  17/25 17 Analysis - Results raw datacompressed (k=32, c=8)

18. VMV 2003  University of Bonn  Computer Graphics Group  18/25 18 Analysis - Results raw datacompressed (k=32, c=8)

19. VMV 2003  University of Bonn  Computer Graphics Group  19/25 19 Real-Time Rendering  Rendering equation for n point-light sources  Evaluate on hardware: closest measured light/view-directions cluster look-up Eigen-BRDF (includes cosine factor)

20. VMV 2003  University of Bonn  Computer Graphics Group  20/25 20 Real-Time Rendering  Straight-Forward GeForce 5900 FX implementation:  ~15 Frames – 800x600, P-IV 2.4GHz  Arranging Eigen-BRDFs in parabolic-maps enables built-in view-interpolation  ~Factor 3 speed-up

21. VMV 2003  University of Bonn  Computer Graphics Group  21/25 21 DemoDemo

22. VMV 2003  University of Bonn  Computer Graphics Group  22/25 22 ExtensionsExtensions  Mip-Mapping:  Assigning weights and cluster-indices to scaled versions of the BTF  Environmental Lighting  Extend “Bi-Scale Radiance Transfer” (Sloan et al. [2003])  Memory savings enable large BTFs  Lighting integral (dot-product of Spherical Harmonics coefficients) could be pre-computed!

23. VMV 2003  University of Bonn  Computer Graphics Group  23/25 23 PreviewPreview

24. VMV 2003  University of Bonn  Computer Graphics Group  24/25 24 ConclusionsConclusions  Using local-PCA for BTF-compression  exploits correlations in the materials structure  especially suited for materials with low- and high- frequency content (high spatial resolution required)  stable fitting algorithm  high quality with affordable memory requirements and runtime cost  implementation on current graphics hardware  easily extendable and combinable with other techniques

25. VMV 2003  University of Bonn  Computer Graphics Group  25/25 25 AcknowledgementsAcknowledgements  Funded by the European Union under the project RealReflect (www.realreflect.org)  Funded by the BMBF under the project VirtualTry-On (www.virtual-try-on.de)  HDR-Environments from www.debevec.org