1. State of the Art in Realtime Ray Tracing & Interactive Global Illumination Ingo Wald Carsten Benthin Joerg Schmittler Philipp Slusallek Saarland University Timothy J. Purcell Stanford University

2. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 2 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

3. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 3 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

4. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 4 Introduction & Motivation Computer Graphics Today: Interactive Graphics (Games etc):  Hardware Accelerated “Triangle Rasterization” Interactive Graphics (Games etc):  Hardware Accelerated “Triangle Rasterization” –“paint each triangles onto the screen” –High performance hardware –But: Only simple models, approximations, only local effects, etc. High-Quality Graphics (Movies, Design, Simulation etc):  Software (offline) “Ray Tracing” High-Quality Graphics (Movies, Design, Simulation etc):  Software (offline) “Ray Tracing” –“trace rays from eye backwards into the scene” –Reverse physical light transport: Can model all effects –All effects, complex scenes, physically correct, etc. –But: Historically too costly for interactivity

5. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 5 Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel

6. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 6 Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene

7. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 7 Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene –Find objects nearby ray (traverse special data structures) –Compute ray-object intersection tests –Determine closest hitpoint Grid (2D)Octree (2D)

8. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 8 Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) Ray Tracing

9. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 9 Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) –Can recursively shoot new rays (e.g. reflections, shadow tests, etc) Ray Tracing

10. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 10 Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) 4.) Display final image Ray Tracing

11. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 11 Ray Tracing Advantages: Complexity is logarithmic in scene size Complexity is logarithmic in scene size  Ideally suited for complex models Allows plug-n-play shading Allows plug-n-play shading  Simple to use and extend Allows for exact visibility queries Allows for exact visibility queries –Physically correct shadows, reflections, global effects,...  Very beneficial to have at interactive rates !

12. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 12 Ray Tracing Advantages: Complexity is logarithmic in scene size Complexity is logarithmic in scene size  Ideally suited for complex models Allows plug-n-play shading Allows plug-n-play shading  Simple to use and extend Allows for exact visibility queries Allows for exact visibility queries –Physically correct shadows, reflections, global effects,...  Very beneficial to have at interactive rates !

14. Part I Interactive Ray Tracing

15. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 15 Realizing Realtime Ray Tracing Realtime Ray Tracing offers many advantages ! Question: How to realize Realtime Ray Tracing ? Today: Three options... Faster software ray tracing Faster software ray tracing –Supercomputers: Muuss et al, Utah [Parker et al.] –Clusters: Saarland OpenRT [Wald et al.], Utah [deMarle et al.] Ray Tracing on programmable GPUs Ray Tracing on programmable GPUs –Carr et al., Purcell et al. Building special purpose hardware Building special purpose hardware –Schmittler et al.

16. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 16 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

17. Software Ray Tracing The RTRT/OpenRT Interactive Ray Tracing System

18. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 18 The OpenRT Interactive Ray Tracing Engine The OpenRT Interactive Ray Tracing Engine: Won’t go into technical details Won’t go into technical details –No details on any kernel algorithms –See the STAR... More important: Summarize “State of the Art” More important: Summarize “State of the Art” –State of the Art of OpenRT Features –State of the Art Performance Data –Today’s possibilities, problems and challenges

19. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 19 The OpenRT Interactive Ray Tracing Engine Features of OpenRT: Highly efficient RT kernels [EG’01,...] Highly efficient RT kernels [EG’01,...] –Specially optimized for modern CPUs (caches, SSE,...) –Recently much faster than EG’01 results... Parallelization on PC cluster [EuroPar’03] Parallelization on PC cluster [EuroPar’03] –Up to 24 dual-Athlon MP 1800+ –Usually linear scalability up to 25 fps@640x480 Can handle dynamic scenes [Part II] Can handle dynamic scenes [Part II] OpenGL-like API [Part II] OpenGL-like API [Part II] –Supports arbitrarily programmable Shaders Supports interactive global illumination [Part III] Supports interactive global illumination [Part III]

20. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 20 RTRT/OpenRT State of the Art Performance Important: Much higher performance than EG’01: Obviously: CPUs got faster.... Obviously: CPUs got faster.... –2.5GHz Pentium IV vs. 866MHz Pentium III: 3x faster Plus: Better compiler support Plus: Better compiler support –Intel C Compiler: ~2x faster than gcc –ICC intrinsics: much better than hand-coded assembler Plus: Better kd tree construction algorithms Plus: Better kd tree construction algorithms –Modified “surface area heuristic” algorithm [Havran]  Roughly 2x faster than originally used kd-trees Plus: Faster implementation... Plus: Faster implementation...  Taken together: Much faster then EG’01

21. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 21 RTRT/OpenRT State of the Art Performance Data State of the art performance data On single PentiumIV 2.5GHz On single PentiumIV 2.5GHz In Frames per second at 1024x1024 pixels In Frames per second at 1024x1024 pixels  Up to 7 million rays per second on 2.5GHz laptop ! RT&Shading RT&ShadingSceneSSE no shd. SSE simple shd. No SSE simple shd. ERW6 (static) 7.12.31.37 ERW6 (dynamic) 4.81.971.06 Conf (static) 4.551.931.2 Conf (dynamic) 2.941.60.82 Soda Hall 4.121.81.055

22. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 22 RTRT/OpenRT Performance Important Lessons Raw ray tracing performance quite high Raw ray tracing performance quite high –Up to 7M rays/sec on 2.5 GHz notebook Corresponds to 7fps@1024x1024Corresponds to 7fps@1024x1024 –Still up to 4 million rays/sec on 1.5Mtri model But: Hard to get peak performance But: Hard to get peak performance –Shading cost, coherent rays,... –But even ‘average’ performance quite high ! Important: Today’s biggest bottleneck is shading ! Important: Today’s biggest bottleneck is shading ! –Even simple shading in ERW6: 7.1Mrps  2.3 Mrps –Future research: Have to look at faster shading ! SSE, Packet shading,...SSE, Packet shading,...

23. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 23 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

24. Ray Tracing on GPUs & Special Purpose Ray Tracing Hardware – The SaarCOR Architecture -

25. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 25 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

26. Part II Advanced Issues

27. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 27 Advanced Ray Tracing Issues Part I: Different ways to realize realtime ray tracing Can’t (yet) say which is best... Can’t (yet) say which is best... But can say: “It is coming” But can say: “It is coming” –Actually, it’s available already today...

28. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 28 Advanced Ray Tracing Issues Ray Tracing is coming  New issues need to be solved: How to handle dynamic scenes ? How to handle dynamic scenes ? –Need new algorithms How can I use it ? How can I use it ?  API ! What new things can I do with it ? What new things can I do with it ? –Build and explore new applications Important: Those issues have to be solved anyway... Important: Those issues have to be solved anyway... –No matter wether it’s GPU’s, CPU’s, or SaarCOR,... –All these issues are equally valid for all architectures!

29. Advanced Ray Tracing Issues: Dynamic Scenes

30. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 30 Handling Dynamic Scenes Ray tracing already beneficial in static scenes Ray tracing already beneficial in static scenes –Visualizing complex models –Complex shading –High quality walkthroughs (e.g. w/ global illumination) –.... But: “Static only” is a severe limitation  Cannot really interact with the model !  Need support for dynamically changing scenes…

31. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 31 Handling Dynamic Scenes Fact: Ray Tracing needs acceleration structure Fact: Ray Tracing needs acceleration structure –Too costly to rebuild each frame Problem: Few research on this topic so far… Problem: Few research on this topic so far… –Just wasn’t interesting before interactive ray tracing… –Previous work: Usually on special cases Utah ‘Hack’: Keep dynamic objects out of accel structure…Utah ‘Hack’: Keep dynamic objects out of accel structure… [Reinhard RW2001]: Incremental updates of Uniform Grid[Reinhard RW2001]: Incremental updates of Uniform Grid – Costly, not hierarchical [Moeller, EG2001]: Only rigid-body animation[Moeller, EG2001]: Only rigid-body animation

32. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 32 Handling Dynamic Scenes Different kinds of dynamic behavior Different kinds of dynamic behavior –Hierarchical, rigid-body motion vs. unstructured motion –Constrained unstructured motion (e.g. maximum displacement) –All triangles animated vs. few triangles animated –Amortized over many or over few rays –… Actually different problems Actually different problems  Need specialized solutions

33. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 33 Handling Dynamic Scenes Our approach: Offer suite of different techniques Offer suite of different techniques –Hierarchical animation of whole objects –Fast reconstruction of objects for unstructured motion (with sacrifices in traversal speed) –High-quality BSPs for often-used static objects (with relatively long rebuild time) Allow to combine them in a hierarchical way Allow to combine them in a hierarchical way  API !

34. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 34 Handling Dynamic Scenes Application groups geometry into ‘objects’ Application groups geometry into ‘objects’ –Similar to building display lists (  API) –Each object has its own acceleration structure Support Unstructured Motion Support Unstructured Motion –Use different BSPs for different objects Cheap BSPs for unstructured motionCheap BSPs for unstructured motion High-quality BSPs for objects w/ rigid-body motionHigh-quality BSPs for objects w/ rigid-body motion –Allow localized, fast rebuild of unstructured motion But: Unstructured motion still costly But: Unstructured motion still costly –Too much data to send over network !!! –Tolerable for moderately complex objects (16k-64ktri)

35. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 35 Handling Dynamic Scenes Objects are ‘instantiated’ into the scene Objects are ‘instantiated’ into the scene –Just like ‘calling’ a display list (  API) –Each instance has a transformation attached to it Hierarchical Anim: Inversely transform rays instead of objectsHierarchical Anim: Inversely transform rays instead of objects  Changing this transformation transforms whole object –Instances are organized in additional hierarchy level With its own acceleration structureWith its own acceleration structure Only this has to be rebuilt every frameOnly this has to be rebuilt every frame Side Effect: Instantiation is for free Side Effect: Instantiation is for free –Sunflowers: 26k instances of 36k-sunflower  1 billion triangles

36. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 36 Handling Dynamic Scenes - Results

37. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 37 Handling Dynamic Scenes - Results TopLevel BSP reconstruction tolerable TopLevel BSP reconstruction tolerable –Some milliseconds even for a few thousand objects –But definitely is not for free... Hierarchical animation is cheap Hierarchical animation is cheap Unstructured motion still a problem Unstructured motion still a problem But: Sufficient for most applications so far But: Sufficient for most applications so far –Used exclusively in all RTRT/OpenRT applications Future work: Future work: –Better support for “immediate mode” rendering –Port algorithms to SaarCOR architecture

38. Advanced Ray Tracing Issues: The OpenRT API

39. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 39 The need for an API So far: All technical problems have been addressed So far: All technical problems have been addressed –Fast ray tracing implementations –Shading, massive models, etc, are easy... –Dynamic scenes  “Theoretically” we’re done... Questions: How can I use it ? Questions: How can I use it ?  Need an API !

40. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 40 The need for an API So far: All technical problems have been addressed So far: All technical problems have been addressed –Fast ray tracing implementations –Shading, massive models, etc, is easy... –Dynamic scenes  “Theoretically” we’re done... Questions: How can I use it ? Questions: How can I use it ?  Need an API ! Problem: No APIs for interactive RT available Problem: No APIs for interactive RT available –OpenGL not suitable for ray tracing –Renderman/Rayshade/Povray inherently offline …

41. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 41 The OpenRT Interactive Ray Tracing API Goals for an Interactive Ray Tracing API: As easy to learn and use as (basic) OpenGL As easy to learn and use as (basic) OpenGL –Leverage existing OpenGL experience As powerful in shading as RenderMan As powerful in shading as RenderMan OpenRT: Combine the best of both Application API much like OpenGL/GLUT Application API much like OpenGL/GLUT –Plus necessary extensions for Ray Tracing Shaders, Objects,...Shaders, Objects,... Shader API like RenderMan Shader API like RenderMan

42. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 42 The OpenRT Interactive Ray Tracing API Application API very OpenGL-like Application API very OpenGL-like –Geometry/Transformations just like OpenGL

43. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 43 The OpenRT Interactive Ray Tracing API Application API very OpenGL-like Application API very OpenGL-like –Geometry/Transformations just like OpenGL rtMatrixMode(RT_MODELVIEW);rtPushMatrix();rtRotatef(1,0,0,M_PI/2);rtScalef(1,2,1);rtBegin(RT_TRIANGLES); rtColor3f(0,1,0) rtColor3f(0,1,0) rtNormal3f(1,1,1); rtNormal3f(1,1,1); rtVertex3f(0,0,0); rtVertex3f(0,0,0);......rtEnd();rtPopMatrix();

44. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 44 The OpenRT Interactive Ray Tracing API Application API very OpenGL-like Application API very OpenGL-like –Geometry/Transformations just like OpenGL –Geometry Objects Just like Display Lists (except: no side effects)Just like Display Lists (except: no side effects) rtNewObjects(), rtBegin/EndObject(), rtInstantiate(),…rtNewObjects(), rtBegin/EndObject(), rtInstantiate(),… –Support fully programmable shader objects Surface, Light, and Pixel Shaders, ‘Renderer Object’Surface, Light, and Pixel Shaders, ‘Renderer Object’ Difference to GL: Different Rendering Semantics Difference to GL: Different Rendering Semantics –“Retained-Mode” vs. “Immediate Mode” Changes take effect only at end of frame !Changes take effect only at end of frame ! No “state machine” modelNo “state machine” model

45. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 45 The OpenRT Interactive Ray Tracing API Shader API Shaders loaded from DLL’s/.so’s Shaders loaded from DLL’s/.so’s Declare Shader Declare Shader –E.g. shader parameters etc Write callback-functions Write callback-functions –‘Shade()’ (for surfaces), ‘Illuminate()’ (for lights), … Access scene data with RenderMan like API Access scene data with RenderMan like API –Geometry: rtsShadingNormal(), … –Lights: rtsIlluminate(), rtsOccluded(),... Can shoot arbitrary secondary rays Can shoot arbitrary secondary rays –rtsTrace(…)

46. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 46 The OpenRT Interactive Ray Tracing API OpenRT Summary OpenRT Summary –Abstracts from underlying engine Completely hides kernel implementationCompletely hides kernel implementation Hides distribution infrastructureHides distribution infrastructure –Low-level API rather than scene graph API –OpenGL like application interface –RenderMan like flexible shader API State of the Art State of the Art –OpenRT V1.0 is out... –Used exclusively to drive Saarland RTRT/OpenRT Used for all shown applications (except GPUs and SaarCOR)Used for all shown applications (except GPUs and SaarCOR) –Ongoing work: SaarCOR@OpenRT SaarCOR@OpenRT

47. Advanced Ray Tracing Issues: Applications

48. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 48 Advanced Ray Tracing Issues: Applications So far: All ingredients for a complete rendering engine are available ! So far: All ingredients for a complete rendering engine are available ! –Technology (Fast ray tracing, dynamic scenes) –API to use it Availability of a new technology opens up potential for new applications Availability of a new technology opens up potential for new applications –Have to explore what’s possible and what not... –Many applications might not even have been thought of ! Here: Only brief overview of today’s possibilities Here: Only brief overview of today’s possibilities

49. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 49 Application I: Classical Ray Tracing Classical Ray Tracing with Complex Shading Effects Classical Ray Tracing with Complex Shading Effects –Local: Volume object, Lightfield, bumpmapping, procedural shaders.... –Global Effects: Shadows, Reflections,...  All effects work together (e.g. shadows from lightfield & volume obj.) “Office” w/ complex shading effects

50. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 50 Application I: Classical Ray Tracing Classical Ray Tracing for physically correct simulation Classical Ray Tracing for physically correct simulation –Industrial project w/ Hella corp –Simulate reflection&refraction behavior of car headlight –Up to 25 levels of reflection/refraction, complex glass effects “Car Headlight” – Physically correct glass simulation

51. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 51 Application II: Visualization of Complex Models Interactive Visualization of Massively Complex Models Interactive Visualization of Massively Complex Models –“Power Plant”: 3x12.5million individual triangles –Simple shading: up to 25 fps... –... on only 2 or 3 clients –Highly complex geometry

52. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 52 Application II: Visualization of Complex Models Oliver Deussen’s “Sunflowers” Scene Oliver Deussen’s “Sunflowers” Scene –Including shadows from sun and transparency through leaves –One billion triangles –7-8 fps @ 24 nodes

53. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 53 Applications III: Interactive Lighting Simulation Lighting Simulation Lighting Simulation –Physically correct simulation of global light propagation Including indirect lighting effectsIncluding indirect lighting effects –Fully recomputed each frame Can move objects, change materials...Can move objects, change materials... –No details here [Part III]

54. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 54 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

55. Part III Using Realtime Ray Tracing for Interactive Global Illumination

56. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 56 Using Realtime Ray Tracing for Interactive G.I. Fast Global Illumination is an old problem... Fast Global Illumination is an old problem... –Lots of previous work Radiosity+OpenGL, Render Cache, Shading Cache, etc.Radiosity+OpenGL, Render Cache, Shading Cache, etc. Usually limited due to lacking performance to (re)compute all effects every frame...Usually limited due to lacking performance to (re)compute all effects every frame... –Need to trace many rays –Ray tracing historically costly  Usually offline Realtime ray tracing: Tracing rays now much faster... Realtime ray tracing: Tracing rays now much faster...  Should theoretically allow for interactive global illumination

57. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 57 Interactive Global Illumination Requirements for Interactive GI Low communication overhead Low communication overhead –Low bandwidth between clients –High latency due to commodity network –No shared memory  Complicates/Forbids use of global data structures Low sampling rate for interactive use Low sampling rate for interactive use –Limited to 40-70 rays (not paths !) per pixel New: Avoid temporal noise and artifacts New: Avoid temporal noise and artifacts –Previously not an issue with off-line methods

58. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 58 Interactive Global Illumination Traditional algorithms: do not scale Traditional algorithms: do not scale –Radiosity Global data structure for radiosity valuesGlobal data structure for radiosity values –Photon Map Global kd-treeGlobal kd-tree Reshooting photons & rebuilding treeReshooting photons & rebuilding tree  Don’t easily map to a RTRT architecture...  MC: would scale but is too noisy MC: would scale but is too noisy –Especially for animations and interactive use !

59. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 59 “Instant Global Illumination” (IGI) Our Approach (w/ Keller and Kollig, Kaiserslautern) Instant Radiosity Technique Instant Radiosity Technique –Smooth lighting approximation using “virtual point lights” Computed with Ray Tracing Computed with Ray Tracing –Adds specular effects (reflections/refraction) –Many coherent shadow rays  Very efficient ! Simplified Photon-Map for Caustics Simplified Photon-Map for Caustics Interleaved Sampling (ILS) Interleaved Sampling (ILS) –Improves overall quality and visual appearance Discontinuity Buffer Discontinuity Buffer –Removes ILS artifacts in image space

60. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 60 IGI: Instant Radiosity Step 1: Trace few particles from lights Step 1: Trace few particles from lights –Place ‘virtual point lights’ (VPLs) at hit points –RQMC techniques (  see Keller’s QMC tutorial notes !!!) Step 2: Illuminate scene from VPLs Step 2: Illuminate scene from VPLs –Per-pixel computation –Supports glossy effects Converges quickly for many VPLs Converges quickly for many VPLs Hard shadow artifacts for few VPLs Hard shadow artifacts for few VPLs

61. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 61 IGI: Interleaved Sampling Use multiple VPL sets per image Use multiple VPL sets per image –Different VPLs for each pixel in a 3x3 grid –Breaks up hard shadow artifacts –Increased number of VPLs per image Still same cost Still same cost Still high coherence Still high coherence –Ray bundle to same set of VPLs Strong Aliasing Strong Aliasing –ILS-Grid is clearly visible

62. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 62 IGI: Discontinuity Buffer Filter Irradiance values Filter Irradiance values –Combine irradiance from neighboring pixels –3x3 grid: using all VPLs for each pixel –Smoothing of ILS-artifacts in image space Adaptive filter Adaptive filter –Avoids filtering across discontinuities (based on normals&distance) –Some Filter artifacts

63. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 63 IGI: Summary Base ingredient Base ingredient –Instant radiosity + fast RT Interleaved sampling Interleaved sampling –Better lighting, coherence Filtering: Remove artifacts Filtering: Remove artifacts –Smoothing in image space Important: All images same #VPLS Important: All images same #VPLS –~Same performance Results: Very good quality Results: Very good quality –Starting at ~20 rays/pixel ! same #VPLs = ~same frame rate !

64. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 64 “IGI2” (2003) Scalable Computation Original implementation [RW’02] had several problems Original implementation [RW’02] had several problems –Scalability problems (server bottleneck,...) –Performance problems (at best 5fps, slow execution per CPU,...) –Quality problems (fixed lighting model, no antialiasing,...)  “IGI2” : New, scalable implementation Still same basic ideas (IR+RT+ILS+DiscoBuffer) Still same basic ideas (IR+RT+ILS+DiscoBuffer) But: New and Improved Ideas But: New and Improved Ideas –No details: Go see the talk tomorrow !

65. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 65 “IGI2” (2003) Scalable Computation New and Improved Ideas [Benthin EG’03, this Friday !] Better Image Quality Better Image Quality –Efficient anti-aliasing 4x-anti-aliasing for 10-20% overhead4x-anti-aliasing for 10-20% overhead –Programmable BRDF-shaders Much faster Much faster –Complete re-implementation –SSE packet tracing of both primary and shadow rays –Parallel shader interface supporting SSE Removed scalability bottleneck Removed scalability bottleneck –Discontinuity filtering on the clients

66. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 66 IGI2 – Scalability Graph Scalability graph on dual processor Athlon 1800+ clients (up to 48 CPUs)

67. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 67 IGI2 Results: Efficient Anti-Aliasing Single sample per pixel at 4 fps (detail view of the conference scene) Four samples per pixel at 3.6 fps  Cost: Only 10% overhead!

68. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 68 IGI2 Results: Performance & Image Quality Old AlgorithmNew Scalable Algorithm Conference Scene on 8 clients at same performance level The new system would render at 12 fps at the same quality as the old system

69. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 69 IGI2 Results: Performance & Image Quality Old AlgorithmNew Scalable Algorithm Conference Scene on 8 clients at same performance level The new system would render at 12 fps at the same quality as the old system

70. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 70 IGI2 – State of the Art

71. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 71 IGI2 – State of the Art

72. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 72 IGI2 – State of the Art

73. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 73 Instant Global Illumination Summary Instant Global Illumination Instant Global Illumination –Instant Radiosity –Interleaved Sampling –Discontinuity Buffering –All realized with fast ray tracing Improved Implementation [EG’03 – This Friday !] Improved Implementation [EG’03 – This Friday !] –Efficient antialiasing –Programmable BRDF shaders –Improved scalability and performance  High performance interactive global illumination –Can compute most important effects at up to 25fps

74. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 74 IGI - Video For more information, also see http://www.OpenRT.de

75. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 75 IGI – Remaining Issues Remaining Problems Many light sources Many light sources –Problem: Too few VPLs for many different lights –Preliminary approach sketched at EGSR’03 Combine contribution estimation and importance samplingCombine contribution estimation and importance sampling Temporal aliasing Temporal aliasing –Especially during changes to the scene... Glossy reflections Glossy reflections Caustics Caustics

76. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 76 Agenda Introduction & Motivation [Wald] Introduction & Motivation [Wald] Part I – Interactive Ray Tracing Architectures [Both] Part I – Interactive Ray Tracing Architectures [Both] –Ray tracing in SW: The RTRT/OpenRT engine [Wald] –Ray tracing on GPUs [Purcell] –Special purpose ray tracing hardware: SaarCOR [Purcell] Part II – Advanced Ray Tracing Issues [Wald] Part II – Advanced Ray Tracing Issues [Wald] –Handling dynamic scenes –The OpenRT interactive ray tracing API –Practical applications Part III – Interactive Global Illumination [Wald] Part III – Interactive Global Illumination [Wald] –Instant Global Illumination Conclusion & Outlook into the Future [Both] Conclusion & Outlook into the Future [Both]

77. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 77 State of the Art in RTRT&IGI Summary Today: Many approaches to realize RTRT Today: Many approaches to realize RTRT –Software systems, GPUs, Special-purpose HW Talked about Advanced Issues Talked about Advanced Issues –Dynamic Scenes, API Issues New applications New applications –Complex Models, high-quality rendering, Interactive Global Illumination,...  Interactive RT has progressed to an advanced state But: Still lots of potential for future research But: Still lots of potential for future research –Better support for dynamic scenes –Better & faster shading

78. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 78 State of the Art in RTRT&IGI Conclusion Realtime Ray Tracing is coming Realtime Ray Tracing is coming –It will soon be realtime on the desktop... faster CPUs/GPUs,faster CPUs/GPUs, better implementationsbetter implementations It will change computer graphics It will change computer graphics –By enabling completely new applications Actually, it is already available today Actually, it is already available today –At least at limited resolutions and frame rate... Go explore your own applications... Go explore your own applications... –Go see what RTRT can do for you...

79. August 9th, 2003Realtime Ray Tracing & Interactive Global Illumination 79 State of the Art in RTRT&IGI Questions ? Questions ?