1. Realistic Reflections and Refractions on Graphics Hardware with Hybrid Rendering and Layered Environment Maps Ziyad S. Hakura John M. Snyder Ziyad S. Hakura John M. Snyder

2. Goal: Real-time rendering of Parameterized Image Spaces including Photorealistic, Reflective/Refractive Objects Goal: Real-time rendering of Parameterized Image Spaces including Photorealistic, Reflective/Refractive Objects Viewpoint Position Object Motion Object Motion

3. Parameterized Image Spaces Space can be 1D, 2D, 3D or more Content author specifies parameters Space can be 1D, 2D, 3D or more Content author specifies parameters Light motion Viewpoint position Object motion

4. Parameterized Image Spaces: Application Examples

5. Animated Mechanism Animated Mechanism viewpoint x time

6. Cockpit Lighting Cockpit Lighting Day light Night sky viewpoint x time of day

7. Interactive Toy Story Limited viewpoint motion  Head motion parallax puts the user “in the scene” Character parameters e.g. happiness/sadness  Rose98, Gleicher98, Popović99 Limited viewpoint motion  Head motion parallax puts the user “in the scene” Character parameters e.g. happiness/sadness  Rose98, Gleicher98, Popović99 viewpoint x time x emotional state

8. Overall Model

12. Ray Tracing vs. Hybrid Rendering Ray Tracing Hybrid Rendering

13. Related Work IBR  Gortler96, Levoy96, Miller98, Wood00 Reflections  Diefenbach96, Ofek98, Cabral99, Lischinski98,  Bastos99 Refractions  Heidrich99, Zongker99, Chuang00 Ray Tracing  Kajiya86, Pharr97 IBR  Gortler96, Levoy96, Miller98, Wood00 Reflections  Diefenbach96, Ofek98, Cabral99, Lischinski98,  Bastos99 Refractions  Heidrich99, Zongker99, Chuang00 Ray Tracing  Kajiya86, Pharr97

14. Our Related Work Parameterized Texture Maps  Hakura00  “pasted on” look away from pre-rendered views Parameterized Environment Maps  Hakura01  view samples must be close together  no refractions Parameterized Texture Maps  Hakura00  “pasted on” look away from pre-rendered views Parameterized Environment Maps  Hakura01  view samples must be close together  no refractions

15. Outline Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions

16. Greedy Ray Path Shading Model N N Reflective Path Reflective Path Refractive Path Refractive Path Trace two ray paths until rays exit object. Refractive Lens Object

17. Greedy Ray Path Shading Model N N Reflective Path Reflective Path Refractive Path Refractive Path Trace two ray paths until rays exit object. Propagate child ray of greatest Fresnel coefficient. Trace two ray paths until rays exit object. Propagate child ray of greatest Fresnel coefficient. Refractive Lens Object

18. Comparison of Shading Models Full binary ray tree Two-term greedy ray path Two-term greedy ray path

19. Outline Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions

20. Impostor Fitting Algorithm For each point in parameter space For each local lens object generate outgoing rays in ray tracer record intersections with environment cluster intersections into layers fit textured impostor to each layer For each point in parameter space For each local lens object generate outgoing rays in ray tracer record intersections with environment cluster intersections into layers fit textured impostor to each layer

21. Layered Environment Maps (EMs)

23. EM Geometry Types Sphere at infinity Sphere Box Ellipsoid Cylinder Quadrilateral Sphere at infinity Sphere Box Ellipsoid Cylinder Quadrilateral

24. Linear Hardware Model A x = b Unknown Texture Pixels Ray-Traced Image HW Filter Coefficients Hardware Render Hardware Render Screen Texture

25. Inferred EMs original image

26. Inferred EMs L 1 cup L 1 cup L 2 cols L 2 cols L 3 walls L 3 walls Reflection Term Reflection Term Refraction Term Refraction Term Inferred Environment Maps

27. Outline Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions

28. Adaptive Tessellation Two criteria:  ray path “topology”  outgoing ray distance Consider both terms of shading model Two criteria:  ray path “topology”  outgoing ray distance Consider both terms of shading model

29. Texture Coordinate Generation

30. Overlaying/Blending Passes L R = L 1 (u R1,v R1 ) over L 2 (u R2,v R2 ) over L 3 (u R3,v R3 ) L T = L 1 (u T1,v T1 ) over L 2 (u T2,v R2 ) over L 3 (u T3,v T3 ) L R = L 1 (u R1,v R1 ) over L 2 (u R2,v R2 ) over L 3 (u R3,v R3 ) L T = L 1 (u T1,v T1 ) over L 2 (u T2,v R2 ) over L 3 (u T3,v T3 ) reflection term refraction term

31. L R = L 1 (u R1,v R1 ) over L 2 (u R2,v R2 ) over L 3 (u R3,v R3 ) L T = L 1 (u T1,v T1 ) over L 2 (u T2,v R2 ) over L 3 (u T3,v T3 ) Single viewpoint (i): L i = L R i F R G R + L T i F T G T Blended viewpoint: L = L i  + L i+1 (1-  ) Requires 4n passes, where n is number of layers L R = L 1 (u R1,v R1 ) over L 2 (u R2,v R2 ) over L 3 (u R3,v R3 ) L T = L 1 (u T1,v T1 ) over L 2 (u T2,v R2 ) over L 3 (u T3,v T3 ) Single viewpoint (i): L i = L R i F R G R + L T i F T G T Blended viewpoint: L = L i  + L i+1 (1-  ) Requires 4n passes, where n is number of layers reflection term refraction term Overlaying/Blending Passes

32. Outline Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions Greedy Ray Path Shading Model Fitting Environment Maps Hybrid Rendering Runtime Results and Conclusions

33. 1D viewspace circling teapot 8º separation between view samples simultaneous solution over 5 viewpoints 3 layers for teapot, 2 for cup 1D viewspace circling teapot 8º separation between view samples simultaneous solution over 5 viewpoints 3 layers for teapot, 2 for cup Experimental Setup

34. Ray-Traced vs. Hybrid Ray-Traced 480 sec/frame Ray-Traced 480 sec/frame Hybrid Rendered 19 sec/frame Hybrid Rendered 19 sec/frame

38. Benefit of Hybrid Rendering over Ray-Tracing Lower cost  Adaptive ray-tracing algorithm Lower cost and higher predictability  Greedy two-term shading model  Environment substitution with layered shells Lower cost  Adaptive ray-tracing algorithm Lower cost and higher predictability  Greedy two-term shading model  Environment substitution with layered shells

39. Future Work Compression with hybrid rendering Handling glossy objects Matching full binary tree renderings More efficient pre-rendering  Multi-dimensional Ray-Tracing Accelerating ray traced animations Making it faster (local ray tracing hardware?) Compression with hybrid rendering Handling glossy objects Matching full binary tree renderings More efficient pre-rendering  Multi-dimensional Ray-Tracing Accelerating ray traced animations Making it faster (local ray tracing hardware?)

40. End

41. Overall Model

42. Greedy Ray Path Shading Model = = result reflection term refraction term * * * * + + T T F T G T R R F R G R

44. Surface Light Fields [Miller98,Wood00] Surface Light Field Dense sampling over surface points of low-resolution lumispheres Dense sampling over surface points of low-resolution lumispheres PEM Sparse sampling over viewpoints of high-resolution EMs Sparse sampling over viewpoints of high-resolution EMs

47. Why Fit Impostors?

48. geometric error of impostor Fitting accounts for:

49. Why Fit Impostors? geometric error of impostor view-dependent shading in environment geometric error of impostor view-dependent shading in environment Fitting accounts for:

50. Why Fit Impostors? geometric error of impostor view-dependent shading in environment geometric error of impostor view-dependent shading in environment Fitting accounts for:

51. simultaneous inference over multiple views  nearby views of lens object  “direct” views (without lens object)  solve confidence-weighted least squares simultaneous inference over multiple views  nearby views of lens object  “direct” views (without lens object)  solve confidence-weighted least squares ray propagation incoming outgoing Preventing Disocclusion Artifacts

52. with without Preventing Disocclusion Artifacts