1. Interactive Rendering using the Render Cache Bruce Walter, George Drettakis iMAGIS*-GRAVIR/IMAG-INRIA Steven Parker University of Utah *iMAGIS is a joint project of CNRS/INRIA/INPG and UJF

2. Motivation Goal: Interactive rendering Ray tracingPath tracing

3. Motivation High-quality renderers Pixel based - Ray tracing, path tracing, etc. Wide range of lighting effects - Reflection, refraction, global illumination, etc. Too slow for interactive use - Many seconds per image - Often used only for final images - Alternate renderers used for interactive editing

4. Motivation Interactive rendering Rapid feedback is paramount - High accuracy is less important Fast consistent framerate - Eg, > 5 fps - Could use faster renderer Eg, hardware accelerated scan conversion Use same renderer - Need to bridge framerate gap

5. Visual Feedback Loop Standard visual feedback loop Entirely synchronous Framerate is limited by the renderer rendererapplication image user

6. Visual Feedback Loop Modified visual feedback loop display application image user renderer Asynchronous interface

7. Goals Independent display process Works with many (pixel-based) renders Exploit frame to frame coherence Reproject pixels from previous frames Fast consistent framerate Use simple, fast methods Concentrate rendering effort Prioritize pixel’s need for recomputation

8. Video

9. Previous Work Approximate or progressive approaches Progressive radiosity or ray tracing Frameless rendering Reprojection or warping Image based rendering (IBR) Ray tracing acceleration

10. Previous Work Parallel processing Multiprocessors or distributed clusters Intelligent display processes Post-rendering warp Holodeck system

11. Algorithm Overview renderer image Render cache interpolate sampling depth cull project Display process

12. Image Estimation Projection Project cached points onto current image - Camera transform provided by application Z-buffer - In case multiple points map to a pixel

13. Image Estimation Problem: visual artifacts Original viewNew view

14. Image Estimation Depth cull heuristic Problem: occluded points may be visible - Z-buffering only works within a pixel Find pixels with locally inconsistent depths - Likely to be from different occluding surfaces Raw projectionAfter depth cull

15. Image Estimation Interpolation / smoothing Problem: small gaps in point data Interpolate pixel colors - Compute locally-weighted average colors - Currently uses 3x3 neighborhoods After interpolationAfter depth cull Raw projection

16. Image Estimation Results after each stage ProjectionDepth cullInterpolation

17. Image Estimation Problem: visual artifacts Simple filter can remove many artifacts Need new points from renderer Previously invisible areas Color changes due to non-diffuse shading - Eg,specular highlights Changes due to user editing - Changes in lighting, geometry, materials

18. Sampling Generate priority image Based on pixel’s need for re-rendering Priority given to pixels with older points - Render cache stores an age with each point Empty pixels priority based on local density - Highest priority given to regions without points

19. Sampling Choose pixels for rendering Sampling must be sparse - Relatively few pixels are rendered per frame Chosen using error-diffusion dither - Concentrates pixels in high priority regions - Maintains good spatial distribution Requested pixels sent to renderer(s) - Results returned at some later frame

20. Sampling Displayed imagePriority imageRequested pixels

21. Optimizations Further prioritizing sampling Identify points that are likely to be outdated - Color change heuristic - Renderer supplied hints Prematurely age these points - Forces sooner resampling of these points

22. Optimizations Moving objects Application can supply object transforms Applied to points in the render cache - Improves tracking of moving objects - Points also aged to encourage resampling

23. Video

24. Results Timing: 70.5 ms or 14 fps 256x256 image, display process only 195 Mhz R10K processor

25. Conclusions Greatly improved interactivity Eg, ray tracing, path tracing Efficient reuse of rendered pixels Using reprojection and simple filters Prioritized sparse sampling Efficently uses limited rendering budget Independent automatic display process Can be used with many different renderers

26. Future Work Larger images Cost scales linearly with # of pixels Higher render ratios Currently work well out to about 1:64 Anti-aliasing

27. The End

28. Overview Cache rendered results Stored as colored points in 3D Estimate current image Project points onto current image plane Filter to reduce artifacts Prioritize future rendering Identify problem pixels Sparse sampling for limited render budget