Originally presented at: ACM SIGGRAPH 2003 Symposium on Interactive 3D Graphics Interruptible Rendering Joint work with: David Luebke * Benjamin Watson† Abhinav Dayal† * University of Virginia † Northwestern University Master’s Project Presentation Cliff Woolley University of Virginia

4/30/2003 Interruptible Rendering2 Motivation

4/30/2003 Interruptible Rendering3 Balance complexity with interactivity  Age-old tradeoff: detail vs. frame rate  Typical approach: Maintain a high, constant frame rate Create a mesh that can always be rendered in time

4/30/2003 Interruptible Rendering4 Improve on traditional LOD  Is a high, constant frame rate good enough?  Is fidelity ever more important than frame rate?  How can this be decided at runtime while still guaranteeing interactivity?

4/30/2003 Interruptible Rendering5 Error metric unification  Inspired by [Meruvia 2000]  Combine notions of temporal and spatial error (i.e. “lateness” vs. “coarseness”)  Use to drive progressive refinement

4/30/2003 Interruptible Rendering6 Unified error metrics  With interruptible rendering, we address this key question: At what point does further refinement of the current frame become pointless?

4/30/2003 Interruptible Rendering7 Methodology

4/30/2003 Interruptible Rendering8 Methodology  Refine a stream of continuous LODs  Monitor input frequently  Minimize dynamic visual error

4/30/2003 Interruptible Rendering9 Refine a stream of continuous LODs  Render progressive refinements on top of each other until “out of time”  Ensure that we can stop refining at any time and move on to the next frame

4/30/2003 Interruptible Rendering10 Refinement  Three refinement schemes implemented: Splatting (Tech sketch, SIGGRAPH 2002) Progressive polygonal hulls Progressive ray casting (Abhinav Dayal, I3D 2003)  Will work with any progressive rendering scheme

4/30/2003 Interruptible Rendering11  Start with a very coarse rendering  Progressively render refinement stream over top of the coarse image Progressive hull refinement

4/30/2003 Interruptible Rendering12 Progressive hulls  Simplification method by [Sander et al. 2000]  Record a sequence of constrained edge collapses and play it back in reverse  Guarantees containment V1V1 VnVn V4V4 V3V3 V2V2

4/30/2003 Interruptible Rendering13 V1V1 VnVn V4V4 V3V3 V2V2 Progressive hulls

4/30/2003 Interruptible Rendering14 Progressive hulls

4/30/2003 Interruptible Rendering15 Monitor input frequently  Ideally: input monitored continuously  Realistically: check every x ms  Allows quick reaction when sudden changes in input occur  Allows system to be self-tuning

4/30/2003 Interruptible Rendering16 Minimize dynamic visual error  Always display image with least error  Sometimes: Further refinement is pointless  Temporal error (lateness) exceeds spatial error (coarseness) for the current frame Front buffer “out of date”  Dynamic visual error (spatial + temporal) in front is greater than in the back

4/30/2003 Interruptible Rendering17 Calculating dynamic visual error  Spatial error Screen-space size of coarsest geometry visible  Temporal error Project bounding box of object in screen-space Find the maximum screen-space distance any corner has moved since frame began

clear front buffer I front = I current start Refine the current image in the front buffer. improve i front t front = I current – I front s front = I front – i front Refine the current image in the back buffer. improve i back t back = I current – I back s back = I back – i back Start refining a new image toward most current input in the back buffer. clear back buffer I back = I current Back buffer now closer to I current than front is. swap buffers I front = I back It is pointless to continue refining. swap buffers I front = I back s front = s back e front >= e back ? noyes no t front > s front ? yesno t back > s back ? Rendering to Back BufferRendering to Front Buffer time t front = I current – I front e front = s front + t front e back = s back + t back Clear the front buffer start Refine the front buffer Refine the back buffer Clear the back buffer Swap buffers e f >= e b ? NY YN t f > s f ? YN t b > s b ? Rendering to Back BufferRendering to Front Buffer time Compute dynamic visual error 

4/30/2003 Interruptible Rendering19 Evaluation

4/30/2003 Interruptible Rendering20 “Gold Standard” evaluation  Compares an ideal rendering to interactive approximations  Ideal rendering: full detail, zero delay  Interactive approximations are Unmanaged Constant fidelity (in pixels) Constant frame rate (fixed Hz) Interruptible rendering

4/30/2003 Interruptible Rendering21 Frame generation  Record what the user sees and when Generate each approximation offline Record actual frames displayed over time  Account for: Render time Double buffering Frame locking Rendering into front buffer (interruptible)

4/30/2003 Interruptible Rendering22 Comparing frames  Error measures RMS – Root Mean Square error Lindstrom's perceptually based error Diff t = Ideal t - Rendered t

4/30/2003 Interruptible Rendering23 Two test input streams  Rotation Model-centered Fixed angular velocity  User interaction Includes both view translation and rotation Includes both static and dynamic segments  Both input streams recorded to files, with timestamps

Ideal Inter- ruptible Un- managed Frames: Interaction sequence: ray caster Constant Fidelity

4/30/2003 Interruptible Rendering25 Video

Interactive input stream Rotation input stream Ray casting Progressive hulls error(rms) time(secs) constant fidelity unmanaged constant frame rate interruptible

4/30/2003 Interruptible Rendering27 Demo

4/30/2003 Interruptible Rendering28 Conclusions

4/30/2003 Interruptible Rendering29 Benefits  Accuracy Balancing spatial and temporal error  Interactivity Slow renderers, like ray caster Large models  Self-tuning feedback system Adapts to hardware

4/30/2003 Interruptible Rendering30 Limitations  Overdraw in progressive renderer Progressive ray caster better here  Cost of monitoring temporal error  Rendering immersive models Requires reversing the containment criteria for polygon rendering Not a problem for ray caster

4/30/2003 Interruptible Rendering31 Future work  Temporal & spatial error Better estimates of each (e.g. color) Better comparisons of both  Textures, normal maps, etc.  Reducing overdraw  View-dependent refinement  Ray tracing: a promising candidate

4/30/2003 Interruptible Rendering32 Acknowledgements  Peter Lindstrom for ltdiff  OpenRT Interactive Raytracing Project  Stanford 3D Scanning Repository  National Science Foundation Awards , , and

4/30/2003 Interruptible Rendering33 Questions?

4/30/2003 Interruptible Rendering34 Progressive ray casting  Progressive refinement Coarse to fine sampling/reconstruction  Sampling Where to shoot the ray  Adaptive  Non-adaptive  Reconstruction Producing imagery from samples

4/30/2003 Interruptible Rendering35 Sampling  Coarse to fine - quadtree approach Sample every quadtree node's center Breadth first traversal Predetermined random traversal per level  Ray casting – using OpenRT Per ray API ~ 600,000 rays per sec. Simple phong shading No shadows or specular highlights

4/30/2003 Interruptible Rendering36 Reconstruction  Placement Place splat at center of each quadtree node  Shading Flat shaded quad covering the node's screen space Alpha-textured quads (smooth reconstruction)  Quad size = twice of node's screen space  Texture: Gaussian blob Transparent at edges Opaque at center

4/30/2003 Interruptible Rendering37 Flat shaded Alpha textured

4/30/2003 Interruptible Rendering38 Calculating dynamic visual error  Temporal error Computed as in case of polygonal hulls  Spatial error Diagonal length of the largest quadtree node displayed on the screen Spatial error