1. Saarbr ü cker IT-Dialog Automobilindustrie A Cave System for Interactive Modeling of Global Illumination in Car Interior Kirill Dmitriev, Thomas Annen, Grzegorz Krawczyk, Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel Max-Planck-Institut für Informatik, Saarbrücken, Germany Karol Myszkowski

2. Saarbr ü cker IT-Dialog Automobilindustrie High Dynamic Range Human eye adjusts comfortably up to 12 orders of magnitude and can see simultaneously up to 4 orders of magnitude

3. Saarbr ü cker IT-Dialog Automobilindustrie HDR Pipeline: Acquisition Global illumination Products: HDR cameras –Lars III (Silicon Vision), Autobrite (SMal Camera Technologies), HDRC (IMS Chips), LM9628 (National), Digital Pixel System (Pixim) Technology [Nayar 2003]: –Trading-off spatial resolution Multiple sensor elements within pixel Spatially varying pixel exposures Mosaicing with spatially varying filter –Trading-off temporal resolution Multi-exposure LDR capture –Multiple image detectors (beam splitters) –Smart pixels

4. Saarbr ü cker IT-Dialog Automobilindustrie HDR Pipeline: Storage Several formats for still images –Radiance –OpenEXR (new industrial standard) –logLuv tiff –HDR JPEG HDR MPEG

5. Saarbr ü cker IT-Dialog Automobilindustrie HDR Pipeline: Display LDR displays: luminance compression required –Solution: tone mapping –Important factors in tone mapping selection Dynamic range of display device Video display conditions –Background lighting Application –Just nice looking video –Visually plausible results –Optimizing visibility of details –Improving contrast … HDR displays start to appear –Sunnybrook Technologies and University of British Columbia

6. Saarbr ü cker IT-Dialog Automobilindustrie HDR Pipeline: Applications IMS Chips

7. Saarbr ü cker IT-Dialog Automobilindustrie Light Reflection emission geometry BRDF radiometric values displayed image Visual DisplayLight Transport Simulation 27 1515 13 22 Display Observer Realistic Image Synthesis Greenberg et al. Siggraph’97. Cornell University HDR LDR

8. Saarbr ü cker IT-Dialog Automobilindustrie Acquisition: Materials Shift variant Bi- directional Reflectance Distribution Function (BRDF) –Lensch et al. Transluscency –Goesele et al.

9. Saarbr ü cker IT-Dialog Automobilindustrie Acquisition: Luminaires Luminaire spatial power distribution (Goesele et al.) –Near field photometry –Emitted energy represented as 4D light field Relevant for light sources installed in the car interior measurement VR: rendering

10. Saarbr ü cker IT-Dialog Automobilindustrie Acquisition: Luminaires Natural lighting –Light probes and multi-exposure techniques –SpheroCam from Spheron VR –Emitted energy represented as an environment map Relevant for external car illumination VR: rendering Light probe Paul Debevec

11. Saarbr ü cker IT-Dialog Automobilindustrie Lighting Simulation + Rendering 1) Photographs of mirror sphere at varying exposure times 2) High-dynamic range environment map 3) Use as light source in Monte Carlo radiosity algorithm Philippe Bekaert

12. Saarbr ü cker IT-Dialog Automobilindustrie Our Goal CAVE system for car interior rendering –Real-time lighting simulation –Dynamic real world lighting HDR video environment maps –Free observer position Head tracking Special focus: –Predict impact of quickly changing lighting conditions on the visibility of information displayed at the LCD panel Precise modeling of light reflections from the LCD panel Predicting effective contrast and information readability for various viewing angles Taking into account light adaptation conditions for the human eye

13. Saarbr ü cker IT-Dialog Automobilindustrie Requirements for Rendering Algorithm 5x2 full screen resolution frames at interactive rates LCD panel illumination must be computed precisely Higher error tolerance for the car interior illumination –Only low spatial frequencies in reconstructed lighting acceptable Distant lighting assumption holds –HDR environment maps acceptable Lighting can change quickly –Relying on temporal coherence between frames impractical Car interior geometry static

14. Saarbr ü cker IT-Dialog Automobilindustrie Rendering Algorithm Selection Requirements –Exploit all computational resources available in the CAVE: 20 CPUs + 10 GPUs Rendering solution –Use Precomputed Radiance Transfer method for car interior: 10 GPUs + 10 CPUs –Use Final Gathering for the LCD panel: 10 CPUs

15. Saarbr ü cker IT-Dialog Automobilindustrie Spherical Harmonics Spherical Harmonics: –Orthonormal basis over the sphere –Analogous to Fourier transform Projection: Reconstruction: Integration:

16. Saarbr ü cker IT-Dialog Automobilindustrie Rendering Equation Incident Light VisibilityCosine Reflected Light Jan Kautz

17. Saarbr ü cker IT-Dialog Automobilindustrie Precomputed Radiance Transfer light function: into SH transfer: transfer: into SH "light vector" "transfer vector" Jan Kautz Such a dot product must be computed for each mesh vertex: n = 25

18. Saarbr ü cker IT-Dialog Automobilindustrie PRT for Arbitrary Meshes Before After

19. Saarbr ü cker IT-Dialog Automobilindustrie Handling Two-Sided Geometry Car geometry is two sided –Need to store SH coefficients for both sides and render geometry twice with back face culling –This leads to lower performance and twice larger memory consumption Better store SH coefficients only for one side of geometry: –Only the rays going through the windows contribute to SH coefficients –User has to point out the mesh parts representing car windows

20. Saarbr ü cker IT-Dialog Automobilindustrie LCD Panel Modeling Emission characteristic of the LCD-sandwich: –Spectral emission-function of the backlight –Transmission characteristic of the polarizer and other layers (e.g. BEF, d-BEF, LCF, rgb-filters, …) –Transmission characteristic of the LC-cell –Transmission of other optical elements – e.g. glass Reflective characteristic in case of incoming light –Surface coating (e.g. AR/AG) –Reflection characteristic of the LCD sandwich observer LCD-sandwich Incoming lighting backlight glassplate

21. Saarbr ü cker IT-Dialog Automobilindustrie Computing LCD Panel Lighting DIMOS or SPECTER systems can be used for lighting simulation within the LCD panel –We use just external spectral emissivity and reflectance data provided with a very high angular resolution Algorithm –Cover geometry of LCD panel with texture –For each texel Compute energy emitted in the observer direction –Use tabulated emissivity data –Modulate emissivity as a function of the displayed information Compute energy reflected in the observer direction –Use the final gathering method –Improve performance using BRDF-weighted importance sampling »If traced ray hits the car window, query the environment map »If traced ray hits the car interior, query PRT data structures

22. Saarbr ü cker IT-Dialog Automobilindustrie System Architecture

23. Saarbr ü cker IT-Dialog Automobilindustrie Distributing the Computation Camera parameters Use “Lightning” system that takes care of OpenGL synchronization between different PCs LCD display computation is distributed in an asynchronous way

24. Saarbr ü cker IT-Dialog Automobilindustrie Distributing the Computation Use “Lightning” system that takes care of OpenGL synchronization between different PCs LCD display computation is distributed in an asynchronous way Broadcast the display image as it is ready

25. Saarbr ü cker IT-Dialog Automobilindustrie Tone Mapping in the CAVE Visual adaptation is influenced by light projected on a small area around the center of retina Head tracking enables precise estimation of the gaze direction –We assume that the adaptation is affected by scene luminances within the region of 10° surrounding the gaze direction –This region can be mapped to 1-3 walls in the CAVE –Luminance data within this region is collected and send to the master Master computes common tone mapping parameters and broadcasts them to all computers in the cluster –There is a small delay in illumination update, but it is hidden by temporal adaptation model anyway

26. Saarbr ü cker IT-Dialog Automobilindustrie Rendering with Tone Mapping 3 Passes: –Compute tone mapping parameters Preview rendering of 128x128 HDR images for the CPU processing –Select visible geometry for tone mapping Lighting OFF Render geometry to z-buffer –Final rendering with tone mapping Lighting ON Use z-buffer test to tone map only visible fragments

27. Saarbr ü cker IT-Dialog Automobilindustrie Results (Car Interior Part) Car model contains approx. 500K triangles Preprocessing of SH coefficients takes about 100 minutes Full model size with SH coefficients is about 60 MB Rendering frame rate is about 10 FPS –For each frame current environment map is projected into SH basis and lighting in every vertex is computed –Full image tone mapping is applied

28. Saarbr ü cker IT-Dialog Automobilindustrie Results (LCD Panel Part) One processor on each PC is busy with sending data to OpenGL Another processor computes draft images (40 samples per pixel) of LCD panel and sends them to other PCs Draft image display is available almost immediately after each head movement or environment maps rotation, converged image is computed in approx. 2 seconds

29. Saarbr ü cker IT-Dialog Automobilindustrie Conclusions We proposed efficient global illumination and tone mapping solutions for CAVE VR applications involving static geometry and dynamic environment lighting We successfully applied those solutions to the car interior modeling, efficiently utilizing all CPUs and GPUs resources on the CAVE cluster We proposed an accurate algorithm for the LCD panel simulation and rendering based on measured BRDF and emission data.

30. Saarbr ü cker IT-Dialog Automobilindustrie Future work Use HDR video stream as dynamic environment map lighting –Trivial to do, we are just waiting for a fish-eye lens suitable for our HDR camera Use HDR display (0.05-3,000 cd/m 2 ) to render the LCD panel with luminance levels similar to real world driving conditions –Display-in-display rendering problem Consider recently proposed techniques for all frequency PRT lighting

31. Saarbr ü cker IT-Dialog Automobilindustrie Acknowledgements We would like to thank the following people –Michael Arnold (Virtual Reality Center, DaimlerChrysler AG) –Thomas Ganz (Research & Technology Displays and Controls (RBP/BM), DaimlerChrysler AG) –Matthias Bues (VR Lab., Fraunhofer IAO)

32. Saarbr ü cker IT-Dialog Automobilindustrie Adaptive Logarithmic Tone Mapping

33. Saarbr ü cker IT-Dialog Automobilindustrie Tone Mapping: Time-Dependent Visual Adaptation 1.Light adaptation 2.Dark adaptation