1. INFORMATIK An Efficient Spatio-Temporal Architecture for Animation Rendering Vlastimil Havran, Cyrille Damez, Karol Myszkowski, and Hans-Peter Seidel Max-Planck-Institut für Informatik, Saarbrücken, Germany Vlastimil Havran, Cyrille Damez, Karol Myszkowski, and Hans-Peter Seidel Max-Planck-Institut für Informatik, Saarbrücken, Germany

2. INFORMATIK Motivation In the traditional approach to rendering of high quality animation sequences every frame is considered separately. The temporal coherence is poorly exploited:The temporal coherence is poorly exploited:  redundant computations. The visual sensitivity to temporal detail cannot be properly accounted for:The visual sensitivity to temporal detail cannot be properly accounted for:  too conservative stopping conditions,  temporal aliasing. In the traditional approach to rendering of high quality animation sequences every frame is considered separately. The temporal coherence is poorly exploited:The temporal coherence is poorly exploited:  redundant computations. The visual sensitivity to temporal detail cannot be properly accounted for:The visual sensitivity to temporal detail cannot be properly accounted for:  too conservative stopping conditions,  temporal aliasing.

3. INFORMATIK Goal Developing an architecture for efficient rendering of high-quality animations which better exploits the spatio-temporal coherence between frames: Visibility: Multi-frame ray tracingVisibility: Multi-frame ray tracing Global illumination: Bi-directional Path Tracing extended to re-using samples between framesGlobal illumination: Bi-directional Path Tracing extended to re-using samples between frames Texturing and shading: sharing information between framesTexturing and shading: sharing information between frames Motion blur: conservative computation in 2D image spaceMotion blur: conservative computation in 2D image space Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories)Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories) Memory requirements: unlimited image resolutionMemory requirements: unlimited image resolution Developing an architecture for efficient rendering of high-quality animations which better exploits the spatio-temporal coherence between frames: Visibility: Multi-frame ray tracingVisibility: Multi-frame ray tracing Global illumination: Bi-directional Path Tracing extended to re-using samples between framesGlobal illumination: Bi-directional Path Tracing extended to re-using samples between frames Texturing and shading: sharing information between framesTexturing and shading: sharing information between frames Motion blur: conservative computation in 2D image spaceMotion blur: conservative computation in 2D image space Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories)Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories) Memory requirements: unlimited image resolutionMemory requirements: unlimited image resolution

4. INFORMATIK Previous Work Industrial solutions such as PhotorealisticRenderman, Maya Rendering System, or SoftimageIndustrial solutions such as PhotorealisticRenderman, Maya Rendering System, or Softimage –frame–by–frame computation –postprocessing of animation to decrease temporal aliasing Industrial solutions such as PhotorealisticRenderman, Maya Rendering System, or SoftimageIndustrial solutions such as PhotorealisticRenderman, Maya Rendering System, or Softimage –frame–by–frame computation –postprocessing of animation to decrease temporal aliasing

5. INFORMATIK Previous Work Temporal Coherence in Ray Tracing Coherence of shaded pixelsCoherence of shaded pixels –Interpolation between frames [Maisel and Hegron 92] –Reprojection [Adelson and Hughes 95] –4D radiance interpolants [Bala et al. 99] Coherence in acceleration data structuresCoherence in acceleration data structures –Space-time solutions [Glassner 88, Groeller 91] –Static vs. dynamic objects [Besuievsky and Pueyo 01, Lext and Moller 01, and Wald et al. 02] Temporal Coherence in Ray Tracing Coherence of shaded pixelsCoherence of shaded pixels –Interpolation between frames [Maisel and Hegron 92] –Reprojection [Adelson and Hughes 95] –4D radiance interpolants [Bala et al. 99] Coherence in acceleration data structuresCoherence in acceleration data structures –Space-time solutions [Glassner 88, Groeller 91] –Static vs. dynamic objects [Besuievsky and Pueyo 01, Lext and Moller 01, and Wald et al. 02]

6. INFORMATIK Previous Work Temporal Coherence in Global Illumination Coherence of illumination samplesCoherence of illumination samples –Render cache [Walter et al. 99] –Shading cache [Tole et al. 02] –Reusing photon hit points [Myszkowski et al. 01] –Selective Photon Tracing [Dmitriev 02] Coherence in visibility computationCoherence in visibility computation –Global lines [Besuievsky and Pueyo 01] Coherence in generation of random numbersCoherence in generation of random numbers –Random number sequence associated with each light transport path [Lafortune 96, Jensen 01, Wald et al. 02] Temporal Coherence in Global Illumination Coherence of illumination samplesCoherence of illumination samples –Render cache [Walter et al. 99] –Shading cache [Tole et al. 02] –Reusing photon hit points [Myszkowski et al. 01] –Selective Photon Tracing [Dmitriev 02] Coherence in visibility computationCoherence in visibility computation –Global lines [Besuievsky and Pueyo 01] Coherence in generation of random numbersCoherence in generation of random numbers –Random number sequence associated with each light transport path [Lafortune 96, Jensen 01, Wald et al. 02]

7. INFORMATIK Space-Time Architecture: Principle Compute samples using a variant of Path-TracingCompute samples using a variant of Path-Tracing Pixels color = mean of sample valuesPixels color = mean of sample values 2 types of samples:2 types of samples: –Native samples:  Expensive  Computed from scratch –Recycled samples:  Cheap  Based on previous computations of native samples (using reprojections) Compute samples using a variant of Path-TracingCompute samples using a variant of Path-Tracing Pixels color = mean of sample valuesPixels color = mean of sample values 2 types of samples:2 types of samples: –Native samples:  Expensive  Computed from scratch –Recycled samples:  Cheap  Based on previous computations of native samples (using reprojections)

8. INFORMATIK Motion Compensation Camera and object motion compensationCamera and object motion compensation Memory access coherence …..Memory access coherence ….. Camera and object motion compensationCamera and object motion compensation Memory access coherence …..Memory access coherence …..

9. INFORMATIK Bi-directional Path Tracing

10. INFORMATIK Camera Motion

11. INFORMATIK Occluded Connection

12. INFORMATIK Path Change

13. INFORMATIK Multi-Frame Ray Tracing Aggregate queriesAggregate queries –Single ray geometry –Results possible for all frames in [f (i-R-S)’ f (i+R) ] Two types of visibility queriesTwo types of visibility queries –Ray shooting –Visibility between two points Three settings of visibility queriesThree settings of visibility queries –Exact result for single frame f (i) –Exact for single frame f (i) + validity for all other frames –Exact results for all frames in [f (i-R)’ f (i+R) ] Aggregate queriesAggregate queries –Single ray geometry –Results possible for all frames in [f (i-R-S)’ f (i+R) ] Two types of visibility queriesTwo types of visibility queries –Ray shooting –Visibility between two points Three settings of visibility queriesThree settings of visibility queries –Exact result for single frame f (i) –Exact for single frame f (i) + validity for all other frames –Exact results for all frames in [f (i-R)’ f (i+R) ]

14. INFORMATIK Multi-Frame Ray Tracing Additional techniques used Mail boxes (cache) for ray transforms, objects, and kd-treesMail boxes (cache) for ray transforms, objects, and kd-trees Frame masks for inserted kd-trees and shadow cacheFrame masks for inserted kd-trees and shadow cache Additional techniques used Mail boxes (cache) for ray transforms, objects, and kd-treesMail boxes (cache) for ray transforms, objects, and kd-trees Frame masks for inserted kd-trees and shadow cacheFrame masks for inserted kd-trees and shadow cache Construction of Spatial Data Structure Instantiation of dynamic objects for range of framesInstantiation of dynamic objects for range of frames Static versus dynamic objects separationStatic versus dynamic objects separation Construction of global kd-tree over static objectsConstruction of global kd-tree over static objects Hierarchical clustering over dynamic objectsHierarchical clustering over dynamic objects Construction of kd-trees for clusters of objectsConstruction of kd-trees for clusters of objects Insertion of kd-trees for clusters into global kd-tree leavesInsertion of kd-trees for clusters into global kd-tree leaves Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods)Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods) Construction of Spatial Data Structure Instantiation of dynamic objects for range of framesInstantiation of dynamic objects for range of frames Static versus dynamic objects separationStatic versus dynamic objects separation Construction of global kd-tree over static objectsConstruction of global kd-tree over static objects Hierarchical clustering over dynamic objectsHierarchical clustering over dynamic objects Construction of kd-trees for clusters of objectsConstruction of kd-trees for clusters of objects Insertion of kd-trees for clusters into global kd-tree leavesInsertion of kd-trees for clusters into global kd-tree leaves Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods)Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods)

15. INFORMATIK The Animation Buffer Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames If more samples are requiredIf more samples are required –Compute a native sample for frame f i –Reproject it and recycle it for all frames in [f (i-R)’ f (i+R) ] S+2R frames are kept in the bufferS+2R frames are kept in the buffer Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames If more samples are requiredIf more samples are required –Compute a native sample for frame f i –Reproject it and recycle it for all frames in [f (i-R)’ f (i+R) ] S+2R frames are kept in the bufferS+2R frames are kept in the buffer SSRRRR

16. INFORMATIK The Animation Buffer Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames If more samples are requiredIf more samples are required –Compute a native sample for frame f i –Reproject it and recycle it for all frames in [f (i-R)’ f (i+R) ] S+2R frames are kept in the bufferS+2R frames are kept in the buffer Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames If more samples are requiredIf more samples are required –Compute a native sample for frame f i –Reproject it and recycle it for all frames in [f (i-R)’ f (i+R) ] S+2R frames are kept in the bufferS+2R frames are kept in the buffer SSRRRRSS Saving to a disk

17. INFORMATIK Shading Computation A simplified version of RenderMan Shading LanguageA simplified version of RenderMan Shading Language Each shader decomposed intoEach shader decomposed into –View-independent component  re-usable, shared between frames –View-dependent component  recomputed for each frame A simplified version of RenderMan Shading LanguageA simplified version of RenderMan Shading Language Each shader decomposed intoEach shader decomposed into –View-independent component  re-usable, shared between frames –View-dependent component  recomputed for each frame

18. INFORMATIK Motion Blur Accuracy & QualityAccuracy & Quality –The same sample point is considered for multiple frames  In other frame-by-frame architectures the motion of objects must be computed explicitely by additional samples. –Temporal changes in shading are properly accounted for  Difficult in other architectures EfficiencyEfficiency –2D computation Accuracy & QualityAccuracy & Quality –The same sample point is considered for multiple frames  In other frame-by-frame architectures the motion of objects must be computed explicitely by additional samples. –Temporal changes in shading are properly accounted for  Difficult in other architectures EfficiencyEfficiency –2D computation

19. INFORMATIK Results Speedup for bi-directional path tracing Moving camera, moving objects: 7.7Moving camera, moving objects: 7.7 Moving objects only: 13.3Moving objects only: 13.3 Moving camera only: 8.8Moving camera only: 8.8 Speedup for bi-directional path tracing Moving camera, moving objects: 7.7Moving camera, moving objects: 7.7 Moving objects only: 13.3Moving objects only: 13.3 Moving camera only: 8.8Moving camera only: 8.8 Proportion of native samples 2.4 - 4.7 % Cost of native samples (profiler) 44 - 64 % of the whole computation time. Disc caching overhead 10 % slowdown (for 1% of main memory used) Time per frame 240 – 415 sec.

20. INFORMATIK Motion blur renderings Motion Blur - computational overhead 25% Time per frame 120 sec. Time per frame 150 sec.

21. INFORMATIK Videos

22. INFORMATIK Conclusions We presented an efficient architecture for rendering of high- quality animationsWe presented an efficient architecture for rendering of high- quality animations In our architecture path tracing, texturing and shading, motion blur can be efficiently computed.In our architecture path tracing, texturing and shading, motion blur can be efficiently computed. Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and camera motion is slow.Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and camera motion is slow. –For more complex motion scenario the animation segment length can be reduced which in the limit may boil down to the traditional frame-by-frame computation. Data structures handling dynamic objects require additional memory, which is acceptable on modern computers.Data structures handling dynamic objects require additional memory, which is acceptable on modern computers. The memory overhead involved in storing multiple images is negligible due to efficient buffering.The memory overhead involved in storing multiple images is negligible due to efficient buffering. We presented an efficient architecture for rendering of high- quality animationsWe presented an efficient architecture for rendering of high- quality animations In our architecture path tracing, texturing and shading, motion blur can be efficiently computed.In our architecture path tracing, texturing and shading, motion blur can be efficiently computed. Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and camera motion is slow.Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and camera motion is slow. –For more complex motion scenario the animation segment length can be reduced which in the limit may boil down to the traditional frame-by-frame computation. Data structures handling dynamic objects require additional memory, which is acceptable on modern computers.Data structures handling dynamic objects require additional memory, which is acceptable on modern computers. The memory overhead involved in storing multiple images is negligible due to efficient buffering.The memory overhead involved in storing multiple images is negligible due to efficient buffering.

23. INFORMATIK Future Work Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002]Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002] Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach.Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach. Skipping some visibility tests based on the spatio- temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et al. 2002]Skipping some visibility tests based on the spatio- temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et al. 2002] Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002]Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002] Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach.Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach. Skipping some visibility tests based on the spatio- temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et al. 2002]Skipping some visibility tests based on the spatio- temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et al. 2002]

24. INFORMATIK Thanks! Polina Kondratieva for rewriting RenderMan ShadersPolina Kondratieva for rewriting RenderMan Shaders Markus Weber for help with preparing the scenesMarkus Weber for help with preparing the scenes Partial support of IST-2001-34744 project RealReflectPartial support of IST-2001-34744 project RealReflect Polina Kondratieva for rewriting RenderMan ShadersPolina Kondratieva for rewriting RenderMan Shaders Markus Weber for help with preparing the scenesMarkus Weber for help with preparing the scenes Partial support of IST-2001-34744 project RealReflectPartial support of IST-2001-34744 project RealReflect All anonymous reviewers for their commentsAll anonymous reviewers for their comments

26. INFORMATIK Space-Time Architecture Rendering based on a variant of path tracing (sample based)Rendering based on a variant of path tracing (sample based) Sample typesSample types –Native samples: computed for a given frame –Recycled samples: re-used for different frames Algorithm:Algorithm: for each pixel // spatial domain while sample variance criterion is met while sample variance criterion is met compute shaded sample; // a sample point in the object space compute shaded sample; // a sample point in the object space for each frame // temporal domain for each frame // temporal domain if possible re-use shaded sample // check visibility and change if possible re-use shaded sample // check visibility and change // sample weight // sample weight Rendering based on a variant of path tracing (sample based)Rendering based on a variant of path tracing (sample based) Sample typesSample types –Native samples: computed for a given frame –Recycled samples: re-used for different frames Algorithm:Algorithm: for each pixel // spatial domain while sample variance criterion is met while sample variance criterion is met compute shaded sample; // a sample point in the object space compute shaded sample; // a sample point in the object space for each frame // temporal domain for each frame // temporal domain if possible re-use shaded sample // check visibility and change if possible re-use shaded sample // check visibility and change // sample weight // sample weight

27. INFORMATIK Space-Time Architecture: Reprojection

28. INFORMATIK Rendering animations – ray tracing with shaders Speedup ray tracing Moving camera, moving objects: 2.6Moving camera, moving objects: 2.6 Speedup ray tracing Moving camera, moving objects: 2.6Moving camera, moving objects: 2.6 ( deterministic reflections must be recomputed! ) ( deterministic reflections must be recomputed! ) Time per frame 770 sec.