Tips and Tricks for Gradient-domain Rendering Implementation

Slides:



Advertisements
Similar presentations
Exploration of advanced lighting and shading techniques
Advertisements

CS123 | INTRODUCTION TO COMPUTER GRAPHICS Andries van Dam © 1/16 Deferred Lighting Deferred Lighting – 11/18/2014.
Isoparametric Elements Element Stiffness Matrices
File Processing - Indirect Address Translation MVNC1 Hashing Indirect Address Translation Chapter 11.
Variance reduction techniques. 2 Introduction Simulation models should be coded such that they are efficient. Efficiency in terms of programming ensures.
Anton S. Kaplanyan Karlsruhe Institute of Technology, Germany Path Space Regularization Framework.
Photorealistic Rendering. Ray tracing v. photorealistic rendering What illumination effects are not captured by ray tracing? What illumination effects.
December 5, 2013Computer Vision Lecture 20: Hidden Markov Models/Depth 1 Stereo Vision Due to the limited resolution of images, increasing the baseline.
Advanced Computer Graphics (Spring 2005) COMS 4162, Lectures 18, 19: Monte Carlo Integration Ravi Ramamoorthi Acknowledgements.
Advanced Computer Graphics (Fall 2009) CS 294, Rendering Lecture 5: Monte Carlo Path Tracing Ravi Ramamoorthi
Final Gathering on GPU Toshiya Hachisuka University of Tokyo Introduction Producing global illumination image without any noise.
Fitting a Model to Data Reading: 15.1,
Advanced Computer Graphics (Spring 2006) COMS 4162, Lecture 20: Monte Carlo Path Tracing Ravi Ramamoorthi Acknowledgements.
1 CE 530 Molecular Simulation Lecture 7 David A. Kofke Department of Chemical Engineering SUNY Buffalo
02/11/05© 2005 University of Wisconsin Last Time Direct Lighting Light Transport Equation (LTE) Intro.
How to do backpropagation in a brain
-Global Illumination Techniques
1 Lesson 8: Basic Monte Carlo integration We begin the 2 nd phase of our course: Study of general mathematics of MC We begin the 2 nd phase of our course:
 Three-Schema Architecture Three-Schema Architecture  Internal Level Internal Level  Conceptual Level Conceptual Level  External Level External Level.
Photo-realistic Rendering and Global Illumination in Computer Graphics Spring 2012 Hybrid Algorithms K. H. Ko School of Mechatronics Gwangju Institute.
- Laboratoire d'InfoRmatique en Image et Systèmes d'information
Emerging Technologies for Games Deferred Rendering CO3303 Week 22.
Photo-realistic Rendering and Global Illumination in Computer Graphics Spring 2012 Hybrid Algorithms K. H. Ko School of Mechatronics Gwangju Institute.
Pure Path Tracing: the Good and the Bad Path tracing concentrates on important paths only –Those that hit the eye –Those from bright emitters/reflectors.
02/12/03© 2003 University of Wisconsin Last Time Intro to Monte-Carlo methods Probability.
Photo-realistic Rendering and Global Illumination in Computer Graphics Spring 2012 Stochastic Path Tracing Algorithms K. H. Ko School of Mechatronics Gwangju.
Slide 1Lastra, 2/14/2016 Monte-Carlo Methods. Slide 2Lastra, 2/14/2016 Topics Kajiya’s paper –Showed that existing rendering methods are approximations.
Real-Time Dynamic Shadow Algorithms Evan Closson CSE 528.
Global Illumination (3) Photon Mapping (1). Overview Light Transport Notation Path Tracing Photon Mapping –Photon Tracing –The Photon Map.
01/26/05© 2005 University of Wisconsin Last Time Raytracing and PBRT Structure Radiometric quantities.
Global Illumination (3) Path Tracing. Overview Light Transport Notation Path Tracing Photon Mapping.
02/9/05© 2005 University of Wisconsin Last Time Lights Next assignment – Implement Kubelka-Munk as a BSDF.
Computer graphics III – Multiple Importance Sampling Jaroslav Křivánek, MFF UK
William Stallings Data and Computer Communications
Lesson 8: Basic Monte Carlo integration
Sampling and Reconstruction of Visual Appearance
Hashing - Hash Maps and Hash Functions
Recent Advances in Light Transport Simulation Theory & Practice
Chapter 11: File System Implementation
Deferred Lighting.
Ray Tracing via Markov Chain Monte-Carlo Method
3D Graphics Rendering PPT By Ricardo Veguilla.
Metropolis light transport
Mike Merchant Nicholas Hilbert
School of Electrical and
Chapter 11: File System Implementation
Optimizing Malloc and Free
© 2005 University of Wisconsin
Primary Sample Space Path Guiding
Simple and Robust Iterative Importance Sampling of Virtual Point Lights Iliyan Georgiev Philipp Slusallek.
Path Tracing (some material from University of Wisconsin)
Chapter 11: File System Implementation
Finite Element Surface-Based Stereo 3D Reconstruction
Advanced Light Transport in the VFX/Archiviz industry
Lecture 29: Virtual Memory-Address Translation
Bidirectional Path Sampling Techniques
Non-parametric Filters: Particle Filters
Chapter 11: File System Implementation
8. One Function of Two Random Variables
Memory System Performance Chapter 3
CS703 - Advanced Operating Systems
Implementation of Learning Systems
Lesson 9: Basic Monte Carlo integration
Future Research.
8. One Function of Two Random Variables
Monte Carlo Path Tracing and Caching Illumination
Path Space Regularization for Robust and Holistic Light Transport
open research problems
Directional Occlusion with Neural Network
Photon Density Estimation using Multiple Importance Sampling
Presentation transcript:

Tips and Tricks for Gradient-domain Rendering Implementation In this section of the course, I will talk about important details and tricks for implementing gradient-domain rendering. As we have a limited amount of time, I will focus on gradient-domain path tracing.

Shift mapping operator The first thing that I want to talk is about shift mapping operator.

PATH TRACING Light Mirror Base Path A typical implementation of a path tracer is to bounce progressively over the scene surface until will hit the light source. This generate a path that connect the sensor to the light source.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Light Mirror Diffuse One of the straightforward strategy to integrate gradient sampling into existing path tracers is to perform shift mapping at each time a new vertex is sampled on the base path. This is the strategy used in the original gradient-domain path tracing implementation. To implement this strategy, you need to track some additional information like the offset path state. Usually, there is 3 different possible path states. Let me quickly enumerate them. Diffuse Diffuse

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector First path state is when the base and offset path are not reconnected. In this state, you need to keep the offset path correlated as much as you can to the base path. For this you can, for example, use half-vector copy. Path state #1: Not reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector If a certain criteria is met, the offset path connect to the base path using explicit reconnection. This state is necessary as re-evaluate the BSDF at the next vertex position as the incoming direction are different between the base and offset path. Path state #2: Just reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector After done the reconnection then the last state consist to keep tracing the base path and offset path together. Path state #3: Reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Implementing a correct shift mapping can be quite challenging. Especially if you deal with Manifold exploration techniques. You need to carefully testing your shift mapping operator by designing experiment to test each case of the shift mapping operator. If you successfully implemented the shift mapping there is another point need that will requires your attention …

Multiple Importance Sampling … the multiple importance sampling.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Indeed, for this pair of path we generated …

Multiple Importance sampling 𝑥 𝑇( 𝑥 ) Forward: … in this way where we got a path X and we shift this base path to the right pixel using the shift mapping operator.

Multiple Importance sampling 𝑥 𝑇( 𝑥 ) 𝑇( 𝑦 ) 𝑦 Forward: Backward: 𝑤 𝑓𝑜𝑟𝑤𝑎𝑟𝑑 𝑥 = 𝑝 𝑥 𝑝 𝑥 +𝑝 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) However, this same pair of path can be generated in the opposite way. As there are two way to generate this given path, we can combine them using multiple importance sampling. To evaluate the MIS weight we need to evaluate the pdf of the base and offset path.

Multiple Importance sampling 𝑤 𝑓𝑜𝑟𝑤𝑎𝑟𝑑 𝑥 = 𝑝 𝑥 𝑝 𝑥 +𝑝 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) Probability can be small Double precisions might be required Might to add epsilon on the denominator Rewrite the weight as a PDF ratio 𝑤 𝑥 = 1 1+ 𝑝 𝑇 𝑥 𝑝( 𝑥 ) 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) However these probability can be small as its express the total probability to arrive to this particular vertex. So inside your practical implementation you want to avoid to have numerical instabilities. To reduce these instabilities, you can use double precision. You can also add a small epsilon to the denominator. This will introduce a small be bias. But it is fine. Finally, you can re-arranging the term inside the MIS weight and storing the ratios of the pdfs. This will mitigate the instability.

HANDLING SHIFT FAILS 1) Shift mapping failed 2) Reversibility failed Need to revert back to finite difference E[G 𝑖 𝑗 ]=𝐸[ 𝑓 𝑖 𝑥 ]− 𝐸[𝑓 𝑗 ( 𝑥 )] Other thing to be carefully with an implementation of gradient-domain is that a lot of times we have to deal with shift fail. For example the shift mapping can failed due to visibility test or due to the fact that the shift is not reversible. To handle shift fails, we need to revert back to the finite difference to estimate the gradients. Handling this change of estimator can be a bit complicated if you need to managed it manually.

Need to revert back to finite difference HANDLING SHIFT FAILS Need to revert back to finite difference E[G 𝑖 𝑗 ]=𝐸[ 𝑓 𝑖 𝑥 ]− 𝐸[𝑓 𝑗 ( 𝑥 )] Simply set the contribution of the offset path to 0 and MIS weight to 1 Extra caution is needed when adaptive sampling is done on the image-space So one trick is to simply return offset path contribution as zero with MIS weight as 1. By doing so, the code revert back to finite difference only when we have some shift mapping failure. This way simplify the implementation. Note that in case of adaptive sampling, extra caution have to be done here as you need to proper normalize this gradient estimator. As the final gradient estimator will contain the gradient estimates using the shift mapping operator or finite difference.

Reducing the cost of shift mapping Another concern when you implement a gradient-domain algorithm is the cost in term of implementation and evaluation of the shift mapping operator.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector 𝑓 𝑥 =0 ~ 𝑓 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) =0 Potentially waste full computation Indeed, with on the fly shift mapping, the base and offset path are constructed at the same time. However, due to some sampling event, the base path can get kill before it bring any contribution on the image plane. Due to the shift mapping design, having zero contribution on the base path means that it is very likely to have a zero contribution on the offset path. So this sampling scheme is potentially not efficient.

CACHED PATHS SHIFT MAPPING Base Path Offset Path Half-vector Generate the base path Apply shift mapping Simplify the code maintance To overcome this issue, we can change how we apply shift mapping. First, we generate the base path. During this generation, we cache some extra information. This information can be geometrical information or which sampling technique have been used to generate this portion of the path and so on… Then, these cached information are used inside the shift mapping operator to generate the offset path. This approach is more general and cleaner as it helps one to separate gradient-domain implementation from the actual path tracing implementation. While there are some small overheads due to extra path storage, it is more manageable when a path tracer becomes complicated.

CACHED PATHS SHIFT MAPPING Base Path Offset Path Half-vector Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 But one benefit of this formulation is that we can decide based on the base path if we want to apply the shift mapping. This decision can be implemented as a Russian roulette decision based on the base path contribution. Applying this Russian roulette scheme originally comes from the path reuse paper. In this formulation, a minimal survival rate of 0.1 is required. Indeed, zero base path contribution does not imply necessary that the offset path will have zero contribution. If you do not have this minimal survival rate, you will have a biased estimator.

CACHED PATHS SHIFT MAPPING G-PT (spp: 32) RR: 56 sec No RR: 89 sec Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 This Russian roulette scheme can speed-up the computation without introducing too much artefacts inside the reconstructed image. This is the results of normal gradient-domain path tracing. You can see that the Russian roulette reduce the rendering time and the reconstructed images looks pretty the same. And …

CACHED PATHS SHIFT MAPPING G-PT with Path Reuse [4x4] (spp: 16) RR: 64 sec No RR: 173 sec Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 … this is the results for path reuse which perform more shift mapping. In the case, this optimization matter as it reduce the rendering time. Notice that this Russian roulette scheme used here is pretty simple. A more elaborate selection on base path where to apply shift mapping can leads to a better performance improvement.

YET ANOTHER SHIFT MAPPING Concerning the shift mapping, if you want to start gradient-domain inside the renderer, you can start first implement this following shift mapping.

YET ANOTHER SHIFT MAPPING Primary Sample Space First you trace the base path. During this tracing you store the random number used.

YET ANOTHER SHIFT MAPPING Primary Sample Space Then, to generate the shift path, you replay the same sequence of random number.

YET ANOTHER SHIFT MAPPING Primary Sample Space Slow Not coherent in difficult scenes Simple implementation No MIS, No shift failed The good thing is that this shift mapping implementation is straightforward as: no MIS required, there is no shift failed and so on. However, this simple approach is slow as it does not reuse the base path. Tracing the offset path have the same cost of retracing a base path. Moreover, as there is no explicit reconnection, the base and the shift path will diverge at some point. As the base and offset path will be not correlated, this leads to an inefficient gradient estimation. Note that, it is possible to remove this issues by using explicit reconnection with this shift mapping. However, in this case, you need to take care again of the MIS computation.

Open-source implementations Finally, let’s talk about the opensource implementation.

Open-source implementations Mitsuba implementation [Jakob 2010] https://github.com/gradientpm/gradient-mts (G-)VPM (G-)BRE (G-)Beam Integrators (Volume) (G-)PT (G-)BDPT (G-)PM (G-)VCM Integrators L1 & L2 Uniform Weighted* Reconstruction *only for G-PT Extra Path Reuse BCD NFOR The first approach implementation is based on Mitsuba. The best way to modify Mitsuba will be to use the bidirectional layer to generate the base path. Then you want to add different shift mapping operator and light transport algorithm on top of that. At the following link you can found the implementation that was using to generate most of the results shown during this course. At your knowledge, this implementation is the most advance one as it contains most of the gradient-domain techniques as: For integration path tracing, directional path tracing, photon mapping and vertex connection and merging and their gradient-domain variants. For integrators for the volume, most of the specialized techniques. For the reconstruction, L1 and L2 reconstruction. As well control variate reconstruction like uniform and weighted reconstruction. Finally, there is path reuse code and other extra stuff like denoising techniques.

Open-source implementations Missing: MCMC Adaptive sampling (Variance-based) More robust reconstruction (Feature-based one) Spectral Temporal (code is available) Mitsuba implementation [Jakob 2010] https://github.com/gradientpm/gradient-mts This implementation is does not cover all the paper in gradient-domain rendering. For example, there is missing some adaptive sampling features, more robust reconstruction and spectral gradient-domain path tracing and temporal gradient-domain path tracing. For the last one, the code is still available on github.

Open-source implementations Aether [Anderson et al. 2017] https://github.com/aekul/aether Yotsuba: https://github.com/aekul/yotsuba Pros: Automatically get PDF, Jacobian, MIS Perfect for prototyping Cons: Slow Not compatible with Manifold exploration Does not handle specular objects Other interesting implementation of gradient-domain path tracing is the work by Anderson and collegue with their domain specific sampling language Aether. The same of authors provide a modified Mistuba version, named Yotsuba, where the gradient-domain path tracer is available. The good thing is this implementation is that PDF evaluation, Jacobian and MIS are given automatically. This remove many difficulties in gradient-domain implementation and perfect for prototyping. However, this features come with a cost like: the code generated is slower, it is impossible to use Manifold exploration and does not handle specular objects.

Open-source implementations Smallgdpt by Tzu-Mao Li Only ~450 lines of code Random replay with explicit reconnection L2 reconstruction Rustlight Written in Rust Different shift mapping Uniform and Weighted reconstruction Finally, two from scratch implementation of gradient-domain path tracing: The first one by Tzu-Mao Li that is small gdpt. The code is very compact and have random replay with explicit reconnection. The authors also provide a L2 reconstruction based on fourier transform. The second one is by us and named rustlight. This implementation is written in Rust language which provide some security about the segfault and add thread safety. Different shift mapping operators are provided and you can use uniform or weighted reconstruction. All these projects are available on Github