Memory-Savvy Distributed Interactive Ray Tracing David E. DeMarle Christiaan Gribble Steven Parker

Impetus for the Paper data sets are growing memory access time is a bottleneck use parallel memory resources efficiently three techniques for faster access to scene data

System Overview base system presented at IEEE PVG’03 cluster port of an interactive ray tracer for shared memory supercomputers IEEE VIS’98 image parallel work division fetch scene data over from peers and cache locally

Three Techniques for Memory Efficiency ODSM  PDSM central work queue  distributed work sharing polygonal mesh reorganization

Distributed Shared Memory data is kept in memory blocks each node has 1/n th of the blocks fetch rest over the network from peers cache recently fetched blocks abstract view of memory 1472 node 1’s memory 2583 node 2’s memory node 3’s memory resident setcache

Object Based DSM each block has a unique handle application finds handle for each datum acquire and release for every block access //locate data handle, offset = ODSM_location(datum); block_start_addr = acquire(handle); //use data datum = *(block_start_addr + offset); //relinquish space release(handle);

ODSM Observations handle = level of indirection  > 4 GB mapping scene data to blocks is tricky acquire and release add overhead address computations add overhead 7.5 GB Richtmyer-Meshkov time step 64 CPUs ~3fps, with view and isovalue changes

Page Based DSM like ODSM: each node keeps 1/n th of scene fetches from peers uses caching difference is how memory is accessed normal virtual memory addressing use addresses between heap and stack PDSM installs a segmentation fault signal handler: on a miss  obtain page from peer, return

PDSM Observations no handles, normal memory access no acquire/release or address computations easy to place any type of scene data in shared space limited to 2^32 bytes  hard to make thread safe  DSM acts only in the exceptional case of a miss ray tracing acceleration structure  > 90 % hit rates ODSMPDSM Hit time10.2 µs4.97 µs Miss time629 µs632 µs

Head-to-Head Comparison compare replication, PDSM and ODSM use a small 512 ^3 volumetric data set PDSM and ODSM keep only 1/16 th locally change viewpoint and isovalue throughout first half, large working set second half, small working set

Head-to-Head Comparison note - accelerated ~2x for presentation

Head-to-Head Comparison

replicated 3.74 frames/sec average

Head-to-Head Comparison ODSM 32% speed of replication

Head-to-Head Comparison PDSM 82% speed of replication

Three Techniques for Memory Efficiency ODSM  PDSM central work queue  distributed work sharing polygonal mesh reorganization

Load Balancing Options central work queue legacy from original shared memory implementation display node keeps task queue render nodes get tiles from queue now  distributed work sharing start with tiles traced last frame  hit rates increase workers get tiles from each other  communicate in parallel, better scalability steal from random peers, slowest worker gives work

Supervisor node tile 0tile 1tile 2tile 3 … Worker node 0 Worker node 1 Worker node 2 Worker node 3 … Worker node 0 Worker node 1 Worker node 2 Worker node 3 … tile 0 tile 1 tile 2 tile 3 … … … … Central Work QueueDistributed Work Sharing

Central Work QueueDistributed Work Sharing

Central Work QueueDistributed Work Sharing

Central Work QueueDistributed Work Sharing

Central Work QueueDistributed Work Sharing

Comparison bunny, dragon, and acceleration structures in PDSM measure misses and frame rates vary local memory to simulate data much larger than physical memory

Misses Frames/Sec 0 1E MB locally E4 central queuedistributed sharing

Misses Frames/Sec 0 1E MB locally E4 central queuedistributed sharing

Misses Frames/Sec 0 1E MB locally E4 central queuedistributed sharing

Misses Frames/Sec 0 1E MB locally E4 central queuedistributed sharing

Three Techniques for Memory Efficiency ODSM  PDSM central work queue  distributed work sharing polygonal mesh reorganization

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &90&91 &2 &92 &3 &93 volume bricking &3&5 &4 &7 &8 &1 &0&2&6 &94&96 &95 &98&90&92 &91&93&97 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &90&91 &2 &92 &3 &93 volume bricking &3&5 &4 &7 &8 &1 &0&2&6 &94&96 &95 &98&90&92 &91&93&97 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &90&91 &2 &92 &3 &93 volume bricking &3&5 &4 &7 &8 &1 &0&2&6 &94&96 &95 &98&90&92 &91&93&97 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &90&91 &2 &92 &3 &93 volume bricking &3&5 &4 &7 &8 &1 &0&2&6 &94&96 &95 &98&90&92 &91&93&97 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &90&91 &2 &92 &3 &93 volume bricking &3&5 &4 &7 &8 &1 &0&2&6 &94&96 &95 &98&90&92 &91&93&97 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &2&3 &4 &6 &5 &7 volume bricking &6&8 &7 &13 &14 &1 &0&2&12 &10&15 &11 &17&3&5 &4&9&16 … … … … mesh “bricking”

Mesh “Bricking” similar to volumetric bricking increase hit rates by reorganizing scene data for better data locality place neighboring triangles on the same page &0&1 &2&3 &4 &6 &5 &7 volume bricking &6&8 &7 &13 &14 &1 &0&2&12 &10&15 &11 &17&3&5 &4&9&16 … … … … mesh “bricking”

Input Mesh

Sorted Mesh

Reorganizing the Mesh based on a grid acceleration structure each grid cell contains pointers to triangles within our grid structure is bricked in memory 1.create grid acceleration structure 2.traverse the cells as stored in memory 3.append copies of the triangles to a new mesh new mesh has triangles sorted in space and memory

Comparison same test as before compare input and sorted mesh

Misses Frames/Sec MB locally input meshsorted mesh

Misses Frames/Sec MB locally input meshsorted mesh

Misses Frames/Sec MB locally input meshsorted mesh

Misses Frames/Sec MB locally input meshsorted mesh

Frames/Sec MB locally input meshsorted mesh grid based approach  duplicates split triangles

Summary three techniques for more efficient memory use: 1.PDSM adds overhead only in the exceptional case of data miss 2.reuse tile assignments with parallel load balancing heuristics 3.mesh reorganization puts related triangles onto nearby pages

Future Work need 64-bit architecture for very large data thread safe PDSM for hybrid parallelism distributed pixel result gathering surface based mesh reorganization

Acknowledgments Funding agencies NSF , DOE VIEWS NIH Reviewers - for tips and seeing through the rough initial data presentation EGPGV Organizers Thank you!