1. Beyond Programmable Shading Course, ACM SIGGRAPH 20111/66

2. Beyond Programmable Shading Course, ACM SIGGRAPH 20112/66 Road to Real-Time Order-Independent Transparency Marco Salvi

3. Beyond Programmable Shading Course, ACM SIGGRAPH 20113/66 Motivation Compositing Equation – Recursive Solvers – Visibility Based Solvers State of the Art and Future Work Q&A Talk Outline

4. Beyond Programmable Shading Course, ACM SIGGRAPH 20114/66 “Reliance on a single program for rendering an entire scene is a poor strategy..” “Separating the image into elements which can be independently rendered saves enormous time..” Why Compositing? “ Compositing Digital Images”, @SIGGRAPH 1984 Thomas Porter & Tom Duff

5. Beyond Programmable Shading Course, ACM SIGGRAPH 20115/66 Motivation “Order- dependent transparency has always been a big limitation for content creators & developers – Restrictive art pipeline: no glass houses – Even windows on cars & buildings can be painful – Restrictive interaction between objects” “Order- independent transparency is must going forward – Big challenge! Gradual process” “ Five Major Challenges in Interactive Rendering”, @SIGGRAPH 2010 Johan Andersson – DICE/EA

6. Beyond Programmable Shading Course, ACM SIGGRAPH 20116/66 Motivation Alpha-Blending Model courtesy of Cem Yuksel.

7. Beyond Programmable Shading Course, ACM SIGGRAPH 20117/66 Motivation Alpha-Blending Model courtesy of Cem Yuksel. Order-Independent Transparency

8. Beyond Programmable Shading Course, ACM SIGGRAPH 20118/66 Motivation

9. Beyond Programmable Shading Course, ACM SIGGRAPH 20119/66 Motivation

10. Beyond Programmable Shading Course, ACM SIGGRAPH 201110/66 Motivation

11. Beyond Programmable Shading Course, ACM SIGGRAPH 201111/66 Motivation Scene courtesy of Valve Corporation. Order-Independent Transparency Alpha-Test

12. Beyond Programmable Shading Course, ACM SIGGRAPH 201112/66 Motivation Order-Independent Transparency Alpha-Test Scene courtesy of Valve Corporation.

13. Beyond Programmable Shading Course, ACM SIGGRAPH 201113/66 Compositing Equation pixel color =

14. Beyond Programmable Shading Course, ACM SIGGRAPH 201114/66 Compositing Equation pixel color = …

15. Beyond Programmable Shading Course, ACM SIGGRAPH 201115/66 Ideal Real-Time OIT Method High image quality – Accurately evaluate compositing equation – No major spatial and temporal artifacts High and stable performance – Low variability. Performance mostly independent from fragments ordering Bounded memory usage – No variable length data structures

16. Beyond Programmable Shading Course, ACM SIGGRAPH 201116/66 Solve compositing equation recursively – Composite fragment with result of previous composite operation Solve compositing equation by computing and evaluating (compressed) visibility functions OIT Algorithms Classification

17. Beyond Programmable Shading Course, ACM SIGGRAPH 201117/66 Alpha Blending / Compositing Depth Peeling A-buffer Z³ algorithm Recursive Solvers

18. Beyond Programmable Shading Course, ACM SIGGRAPH 201118/66 Alpha Compositing [Porter and Duff 1984]

19. Beyond Programmable Shading Course, ACM SIGGRAPH 201119/66 Alpha Compositing [Porter and Duff 1984]

20. Beyond Programmable Shading Course, ACM SIGGRAPH 201120/66 Alpha Compositing [Porter and Duff 1984]

21. Beyond Programmable Shading Course, ACM SIGGRAPH 201121/66 Alpha Compositing [Porter and Duff 1984]

22. Beyond Programmable Shading Course, ACM SIGGRAPH 201122/66 Alpha Compositing [Porter and Duff 1984]

23. Beyond Programmable Shading Course, ACM SIGGRAPH 201123/66 Alpha Compositing [Porter and Duff 1984] Fast and stable/predictable performance No additional storage required But order- dependent..

24. Beyond Programmable Shading Course, ACM SIGGRAPH 201124/66 Peel layers and composite in depth- sorted order Number of passes proportional to max image depth complexity Depth Peeling [Everitt 2001][Liu et al. 2009] EVERITT, C. 2001. Interactive order-independent transparency. Tech. rep.

25. Beyond Programmable Shading Course, ACM SIGGRAPH 201125/66 1) Render fragments color and depth in per-pixel lists A-buffer [Carpenter 1984] [Yang et al. 2009]

26. Beyond Programmable Shading Course, ACM SIGGRAPH 201126/66 1) Render fragments color and depth in per-pixel lists 2) Per-pixel sort and composite fragments A-buffer [Carpenter 1984] [Yang et al. 2009]

27. Beyond Programmable Shading Course, ACM SIGGRAPH 201127/66 1) Render fragments color and depth in per-pixel lists 2) Per-pixel sort and composite fragments A-buffer [Carpenter 1984] [Yang et al. 2009] to the frame buffer

28. Beyond Programmable Shading Course, ACM SIGGRAPH 201128/66 A-buffer Limitations Poor & unstable performance, tipically memory bandwidth limited Unbounded memory requirements Scene courtesy of Valve Corporation.

29. Beyond Programmable Shading Course, ACM SIGGRAPH 201129/66 Bounded A-buffer – Up to N fragments per pixel (sorted) Merge fragments to keep pixel memory footprint constant – Distance & coverage based compression metric The Z³ algorithm [Jouppi and Chang 1999] JOUPPI, N. P., AND CHANG, C.-F. 1999. Z3: an economical hard- ware technique for high-quality antialiasing and transparency.

30. Beyond Programmable Shading Course, ACM SIGGRAPH 201130/66 Visibility Function Models light absorption Product of step functions (thin blockers)

31. Beyond Programmable Shading Course, ACM SIGGRAPH 201131/66 Fragment Parallel Compositing Occupancy Maps Opacity Shadow Methods Stochastic Transparency Adaptive Transparency Visibility Based Solvers

32. Beyond Programmable Shading Course, ACM SIGGRAPH 201132/66 Compute visibility via parallel segmented scan – Load-balance across irregular number of fragments per pixel Evaluate compositing equation via parallel reduction Frag. Parallel Compositing [Patney et al. 2010] pixel color = …

33. Beyond Programmable Shading Course, ACM SIGGRAPH 201133/66 Assume all transp. geom. has the same alpha – Not applicable to many objects (smoke, foliage, etc.) Bit-field represents step functions of same height and spaced at regular depth – A slab map allows to re-modulate steps height on a given depth region Use visibility representation for OIT & shadows Occupancy Maps [Sintorn et al. 2009]

34. Beyond Programmable Shading Course, ACM SIGGRAPH 201134/66 Replace visibility with opacity: – Less well-behaved function (no monotonicity, no [0,1] bounds) Basis function – Piece-wise constant intervals: Opacity Shadow Maps [Kim et al. 2001] Warp OSMs first layer: Deep Opacity Shadow Maps [Yuksel et al. 2008] – Trigonometric: Fourier Opacity Mapping [Jansen et al. 2010] Opacity Mapping Methods

35. Beyond Programmable Shading Course, ACM SIGGRAPH 201135/66 Fixed length visibility representation – Regular steps at irregular locations – A2C & z-test on MSAA samples Compression by removing random step Efficient use GPU fixed function HW Fast but “noisy” with complex objs – Can require many samples per-pixel Stochastic Transparency [Enderton et al. 2010] ENDERTON, E., SINTORN, E., SHIRLEY, P., AND LUEBKE, D. 2010. Stochastic transparency.

36. Beyond Programmable Shading Course, ACM SIGGRAPH 201136/66 Derived from volumetric shadow method [Salvi et al. 2010] – Optimized for thin objs and OIT Store up to N steps per pixel – Arbitrary location and height – Area based compression metric Adaptive Transparency [Salvi et al. 2011] Salvi M., Montgomery J., Lefohn A. 2001. Adaptive Transparency

37. Beyond Programmable Shading Course, ACM SIGGRAPH 201137/66 To compress visibility AT removes the node that generates the smallest area variation Adaptive Transparency [Salvi et al. 2011]

38. Beyond Programmable Shading Course, ACM SIGGRAPH 201138/66 To compress visibility AT removes the node that generates the smallest area variation Adaptive Transparency [Salvi et al. 2011]

39. Beyond Programmable Shading Course, ACM SIGGRAPH 201139/66 To compress visibility AT removes the node that generates the smallest area variation Adaptive Transparency [Salvi et al. 2011]

40. Beyond Programmable Shading Course, ACM SIGGRAPH 201140/66 To compress visibility AT removes the node that generates the smallest area variation Adaptive Transparency [Salvi et al. 2011]

41. Beyond Programmable Shading Course, ACM SIGGRAPH 201141/66 To compress visibility AT removes the node that generates the smallest area variation Adaptive Transparency [Salvi et al. 2011]

42. Beyond Programmable Shading Course, ACM SIGGRAPH 201142/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

43. Beyond Programmable Shading Course, ACM SIGGRAPH 201143/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

44. Beyond Programmable Shading Course, ACM SIGGRAPH 201144/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

45. Beyond Programmable Shading Course, ACM SIGGRAPH 201145/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

46. Beyond Programmable Shading Course, ACM SIGGRAPH 201146/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

47. Beyond Programmable Shading Course, ACM SIGGRAPH 201147/66 To compress visibility AT removes the node that generates the smallest area variation smallest area variation Adaptive Transparency [Salvi et al. 2011]

48. Beyond Programmable Shading Course, ACM SIGGRAPH 201148/66 Adaptive Transparency [Salvi et al. 2011] FOREST scene 8 ms - 6.0 MFragment Max fragment per pixel: 45 7x faster than A-buffer 2x faster than Stoc. Transp. SMOKE scene 21 ms - 10.6 MFragment Max fragment per pixel: 312 30x faster than A-buffer 2.5x faster than Stoc. Transp. HAIR scene 48 ms - 15.0 MFragment Max fragment per pixel: 663 40x faster than A-buffer 2x faster than Stoc. Transp. Scene courtesy of Valve Corporation. Model courtesy of Cem Yuksel. Scene courtesy of Valve Corporation.

49. Beyond Programmable Shading Course, ACM SIGGRAPH 201149/66 Higher quality than other “lossy” OIT methods – Works on any transparent object type (foliage, smoke, glass, etc.) Designed to run in fixed memory – Prototype uses variable memory data struct due to 3D APIs limitations High and scalable performance – Easy to trade-off IQ for performance and storage by tuning per-pixel step/node count Adaptive Transparency [Salvi et al. 2011]

50. Beyond Programmable Shading Course, ACM SIGGRAPH 201150/66 AT & ST are good examples of new algorithms close to ideal OIT method for real-time apps – ST main limitation is noise / lower quality per vis. rep. storage – AT main limitation is current variable memory implementation We are rapidly converging towards entirely practical and robust OIT methods for real-time rendering! What’s Next?

51. Beyond Programmable Shading Course, ACM SIGGRAPH 201151/66 Deferred shading and OIT methods – Many methods are trivial to extend to DSers.. –..but awfully inefficient.. Some interesting work left to do in this area Open Problems

52. Beyond Programmable Shading Course, ACM SIGGRAPH 201152/66 Acknowledgements Special thanks to Jefferson Montgomery and Aaron Lefohn Craig Kolb, Matt Pharr, Charles Lingle and Elliot Garbus for supporting this work. Jason Mitchell and Wade Schinn at Valve Software for the assets from Left- for-Dead-2 Cem Yuksel (Cornell University) for the hair model

53. Beyond Programmable Shading Course, ACM SIGGRAPH 201153/66 Q&A “Adaptive Transparency” pre-print: http://intel.ly/at_hpg11http://intel.ly/at_hpg11 “Adaptive Transparency” source code: http://intel.ly/aoit_gdchttp://intel.ly/aoit_gdc twitter: @marcosalvi e-mail: marco.salvi@intel.commarco.salvi@intel.com blog: http://pixelstoomany.wordpress.comhttp://pixelstoomany.wordpress.com

54. Beyond Programmable Shading Course, ACM SIGGRAPH 201154/66 Backup

55. Beyond Programmable Shading Course, ACM SIGGRAPH 201155/66BibliographyBibliography PORTER, T., AND DUFF, T. 1984. “Compositing digital images.” SIGGRAPH Comput. Graph. 18, 3, 253– 259. CARPENTER, L. 1984. “The A -buffer, an antialiased hidden surface method.” SIGGRAPH Comput. Graph. 18, 3, 103–108. SINTORN, E., AND ASSARSON, U. 2009. “Hair self shadowing and transparency depth ordering using occupancy maps.” In I3D ’09: Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games, 67–74 PATNEY, A., TZENG, S., AND OWENS, J. D. 2010. “Fragment parallel composite and filter.” Computer Graphics Forum (Proceedings of EGSR 2010) 29, 4 (June), 1251–1258. YANG, J., HENSLEY, J., GR¨U 547 N, H., AND THIBIEROZ, N. 2010. “Real-time concurrent linked list construction on the gpu.” Computer Graphics Forum (Proceedings of EGSR 2010) 29, 4 (June), SALVI, M., VIDIMCE, K., LAURITZEN, A., AND LEFOHN, A. 2010. “Adaptive volumetric shadow maps.” Computer Graphics Forum (Proceedings of EGSR 2010) 29, 4 (June), 1289–1296.

56. Beyond Programmable Shading Course, ACM SIGGRAPH 201156/66 Idea: Save bandwidth by working with an approximate visibility function 200+ steps

57. Beyond Programmable Shading Course, ACM SIGGRAPH 201157/66 Idea: Save bandwidth by working with an approximate visibility function 200+ steps

58. Beyond Programmable Shading Course, ACM SIGGRAPH 201158/66 Idea: Save bandwidth by working with an approximate visibility function 32 steps 200+ steps

59. Beyond Programmable Shading Course, ACM SIGGRAPH 201159/66 Idea: Save bandwidth by working with an approximate visibility function 32 steps 200+ steps

60. Beyond Programmable Shading Course, ACM SIGGRAPH 201160/66 Idea: Save bandwidth by working with an approximate visibility function 32 steps 200+ steps

61. Beyond Programmable Shading Course, ACM SIGGRAPH 201161/66 AT GPU Implementation Store visibility in the frame buffer? – Data update cannot be mapped to DX11 blend modes – No RMW operations on the frame buffer Store visibility in a Read/Write buffer (UAV)? – Cause data races

62. Beyond Programmable Shading Course, ACM SIGGRAPH 201162/66 AT Proof-of-Concept Implementation 1) Render transparent fragments to per-pixel lists – Same as A-buffer implementation 2) For each pixel: build an approximate visibility function and use it to composite all transparent fragments – Full-screen pass guarantees atomicity

63. Beyond Programmable Shading Course, ACM SIGGRAPH 201163/66 Bandwidth Requirements

64. Beyond Programmable Shading Course, ACM SIGGRAPH 201164/66 AT Future Work Investigate bounded memory implementations – Per-pixel locks? New frame-buffer format? Better visibility data compression – Reduce MSAA impact on memory requirements

65. Beyond Programmable Shading Course, ACM SIGGRAPH 201165/66 Scene courtesy of Valve Corporation.

66. Beyond Programmable Shading Course, ACM SIGGRAPH 201166/66 Capture fragments in FIFO – Enable efficient multi-pass shading Require global sort – A-buffer requires local sort FIFO can overflow – Similar to A-buffer F-Buffer [Mark and Proudfoot 2001] MARK, W. R., AND PROUDFOOT, K. 2001. The F-buffer: A rasterization-order FIFO buffer for multi-pass rendering