Tips and Tricks for Gradient-domain Rendering Implementation In this section of the course, I will talk about important details and tricks for implementing gradient-domain rendering. As we have a limited amount of time, I will focus on gradient-domain path tracing.

Shift mapping operator The first thing that I want to talk is about shift mapping operator.

PATH TRACING Light Mirror Base Path A typical implementation of a path tracer is to bounce progressively over the scene surface until will hit the light source. This generate a path that connect the sensor to the light source.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Light Mirror Diffuse One of the straightforward strategy to integrate gradient sampling into existing path tracers is to perform shift mapping at each time a new vertex is sampled on the base path. This is the strategy used in the original gradient-domain path tracing implementation. To implement this strategy, you need to track some additional information like the offset path state. Usually, there is 3 different possible path states. Let me quickly enumerate them. Diffuse Diffuse

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector First path state is when the base and offset path are not reconnected. In this state, you need to keep the offset path correlated as much as you can to the base path. For this you can, for example, use half-vector copy. Path state #1: Not reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector If a certain criteria is met, the offset path connect to the base path using explicit reconnection. This state is necessary as re-evaluate the BSDF at the next vertex position as the incoming direction are different between the base and offset path. Path state #2: Just reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector After done the reconnection then the last state consist to keep tracing the base path and offset path together. Path state #3: Reconnected

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Implementing a correct shift mapping can be quite challenging. Especially if you deal with Manifold exploration techniques. You need to carefully testing your shift mapping operator by designing experiment to test each case of the shift mapping operator. If you successfully implemented the shift mapping there is another point need that will requires your attention …

Multiple Importance Sampling … the multiple importance sampling.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector Indeed, for this pair of path we generated …

Multiple Importance sampling 𝑥 𝑇( 𝑥 ) Forward: … in this way where we got a path X and we shift this base path to the right pixel using the shift mapping operator.

Multiple Importance sampling 𝑥 𝑇( 𝑥 ) 𝑇( 𝑦 ) 𝑦 Forward: Backward: 𝑤 𝑓𝑜𝑟𝑤𝑎𝑟𝑑 𝑥 = 𝑝 𝑥 𝑝 𝑥 +𝑝 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) However, this same pair of path can be generated in the opposite way. As there are two way to generate this given path, we can combine them using multiple importance sampling. To evaluate the MIS weight we need to evaluate the pdf of the base and offset path.

Multiple Importance sampling 𝑤 𝑓𝑜𝑟𝑤𝑎𝑟𝑑 𝑥 = 𝑝 𝑥 𝑝 𝑥 +𝑝 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) Probability can be small Double precisions might be required Might to add epsilon on the denominator Rewrite the weight as a PDF ratio 𝑤 𝑥 = 1 1+ 𝑝 𝑇 𝑥 𝑝( 𝑥 ) 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) However these probability can be small as its express the total probability to arrive to this particular vertex. So inside your practical implementation you want to avoid to have numerical instabilities. To reduce these instabilities, you can use double precision. You can also add a small epsilon to the denominator. This will introduce a small be bias. But it is fine. Finally, you can re-arranging the term inside the MIS weight and storing the ratios of the pdfs. This will mitigate the instability.

HANDLING SHIFT FAILS 1) Shift mapping failed 2) Reversibility failed Need to revert back to finite difference E[G 𝑖 𝑗 ]=𝐸[ 𝑓 𝑖 𝑥 ]− 𝐸[𝑓 𝑗 ( 𝑥 )] Other thing to be carefully with an implementation of gradient-domain is that a lot of times we have to deal with shift fail. For example the shift mapping can failed due to visibility test or due to the fact that the shift is not reversible. To handle shift fails, we need to revert back to the finite difference to estimate the gradients. Handling this change of estimator can be a bit complicated if you need to managed it manually.

Need to revert back to finite difference HANDLING SHIFT FAILS Need to revert back to finite difference E[G 𝑖 𝑗 ]=𝐸[ 𝑓 𝑖 𝑥 ]− 𝐸[𝑓 𝑗 ( 𝑥 )] Simply set the contribution of the offset path to 0 and MIS weight to 1 Extra caution is needed when adaptive sampling is done on the image-space So one trick is to simply return offset path contribution as zero with MIS weight as 1. By doing so, the code revert back to finite difference only when we have some shift mapping failure. This way simplify the implementation. Note that in case of adaptive sampling, extra caution have to be done here as you need to proper normalize this gradient estimator. As the final gradient estimator will contain the gradient estimates using the shift mapping operator or finite difference.

Reducing the cost of shift mapping Another concern when you implement a gradient-domain algorithm is the cost in term of implementation and evaluation of the shift mapping operator.

ON THE FLY SHIFT MAPPING Base Path Offset Path Half-vector 𝑓 𝑥 =0 ~ 𝑓 𝑇 𝑥 𝑑𝜇(𝑇 𝑥 ) 𝑑𝜇( 𝑥 ) =0 Potentially waste full computation Indeed, with on the fly shift mapping, the base and offset path are constructed at the same time. However, due to some sampling event, the base path can get kill before it bring any contribution on the image plane. Due to the shift mapping design, having zero contribution on the base path means that it is very likely to have a zero contribution on the offset path. So this sampling scheme is potentially not efficient.

CACHED PATHS SHIFT MAPPING Base Path Offset Path Half-vector Generate the base path Apply shift mapping Simplify the code maintance To overcome this issue, we can change how we apply shift mapping. First, we generate the base path. During this generation, we cache some extra information. This information can be geometrical information or which sampling technique have been used to generate this portion of the path and so on… Then, these cached information are used inside the shift mapping operator to generate the offset path. This approach is more general and cleaner as it helps one to separate gradient-domain implementation from the actual path tracing implementation. While there are some small overheads due to extra path storage, it is more manageable when a path tracer becomes complicated.

CACHED PATHS SHIFT MAPPING Base Path Offset Path Half-vector Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 But one benefit of this formulation is that we can decide based on the base path if we want to apply the shift mapping. This decision can be implemented as a Russian roulette decision based on the base path contribution. Applying this Russian roulette scheme originally comes from the path reuse paper. In this formulation, a minimal survival rate of 0.1 is required. Indeed, zero base path contribution does not imply necessary that the offset path will have zero contribution. If you do not have this minimal survival rate, you will have a biased estimator.

CACHED PATHS SHIFT MAPPING G-PT (spp: 32) RR: 56 sec No RR: 89 sec Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 This Russian roulette scheme can speed-up the computation without introducing too much artefacts inside the reconstructed image. This is the results of normal gradient-domain path tracing. You can see that the Russian roulette reduce the rendering time and the reconstructed images looks pretty the same. And …

CACHED PATHS SHIFT MAPPING G-PT with Path Reuse [4x4] (spp: 16) RR: 64 sec No RR: 173 sec Russian roulette (RR): 𝑞=max min 1, lum 𝑓 𝑥 0.1 ,0.1 … this is the results for path reuse which perform more shift mapping. In the case, this optimization matter as it reduce the rendering time. Notice that this Russian roulette scheme used here is pretty simple. A more elaborate selection on base path where to apply shift mapping can leads to a better performance improvement.

YET ANOTHER SHIFT MAPPING Concerning the shift mapping, if you want to start gradient-domain inside the renderer, you can start first implement this following shift mapping.

YET ANOTHER SHIFT MAPPING Primary Sample Space First you trace the base path. During this tracing you store the random number used.

YET ANOTHER SHIFT MAPPING Primary Sample Space Then, to generate the shift path, you replay the same sequence of random number.

YET ANOTHER SHIFT MAPPING Primary Sample Space Slow Not coherent in difficult scenes Simple implementation No MIS, No shift failed The good thing is that this shift mapping implementation is straightforward as: no MIS required, there is no shift failed and so on. However, this simple approach is slow as it does not reuse the base path. Tracing the offset path have the same cost of retracing a base path. Moreover, as there is no explicit reconnection, the base and the shift path will diverge at some point. As the base and offset path will be not correlated, this leads to an inefficient gradient estimation. Note that, it is possible to remove this issues by using explicit reconnection with this shift mapping. However, in this case, you need to take care again of the MIS computation.

Open-source implementations Finally, let’s talk about the opensource implementation.

Open-source implementations Mitsuba implementation [Jakob 2010] https://github.com/gradientpm/gradient-mts (G-)VPM (G-)BRE (G-)Beam Integrators (Volume) (G-)PT (G-)BDPT (G-)PM (G-)VCM Integrators L1 & L2 Uniform Weighted* Reconstruction *only for G-PT Extra Path Reuse BCD NFOR The first approach implementation is based on Mitsuba. The best way to modify Mitsuba will be to use the bidirectional layer to generate the base path. Then you want to add different shift mapping operator and light transport algorithm on top of that. At the following link you can found the implementation that was using to generate most of the results shown during this course. At your knowledge, this implementation is the most advance one as it contains most of the gradient-domain techniques as: For integration path tracing, directional path tracing, photon mapping and vertex connection and merging and their gradient-domain variants. For integrators for the volume, most of the specialized techniques. For the reconstruction, L1 and L2 reconstruction. As well control variate reconstruction like uniform and weighted reconstruction. Finally, there is path reuse code and other extra stuff like denoising techniques.

Open-source implementations Missing: MCMC Adaptive sampling (Variance-based) More robust reconstruction (Feature-based one) Spectral Temporal (code is available) Mitsuba implementation [Jakob 2010] https://github.com/gradientpm/gradient-mts This implementation is does not cover all the paper in gradient-domain rendering. For example, there is missing some adaptive sampling features, more robust reconstruction and spectral gradient-domain path tracing and temporal gradient-domain path tracing. For the last one, the code is still available on github.

Open-source implementations Aether [Anderson et al. 2017] https://github.com/aekul/aether Yotsuba: https://github.com/aekul/yotsuba Pros: Automatically get PDF, Jacobian, MIS Perfect for prototyping Cons: Slow Not compatible with Manifold exploration Does not handle specular objects Other interesting implementation of gradient-domain path tracing is the work by Anderson and collegue with their domain specific sampling language Aether. The same of authors provide a modified Mistuba version, named Yotsuba, where the gradient-domain path tracer is available. The good thing is this implementation is that PDF evaluation, Jacobian and MIS are given automatically. This remove many difficulties in gradient-domain implementation and perfect for prototyping. However, this features come with a cost like: the code generated is slower, it is impossible to use Manifold exploration and does not handle specular objects.

Open-source implementations Smallgdpt by Tzu-Mao Li Only ~450 lines of code Random replay with explicit reconnection L2 reconstruction Rustlight Written in Rust Different shift mapping Uniform and Weighted reconstruction Finally, two from scratch implementation of gradient-domain path tracing: The first one by Tzu-Mao Li that is small gdpt. The code is very compact and have random replay with explicit reconnection. The authors also provide a L2 reconstruction based on fourier transform. The second one is by us and named rustlight. This implementation is written in Rust language which provide some security about the segfault and add thread safety. Different shift mapping operators are provided and you can use uniform or weighted reconstruction. All these projects are available on Github