Download presentation
Presentation is loading. Please wait.
1
Inverse Volume Rendering with Material Dictionaries
Ioannis Gkioulekas1 Shuang Zhao2 Kavita Bala2 Todd Zickler1 Anat Levin3 1Harvard 2Cornell 3Weizmann
2
Most materials are translucent
food skin jewelry architecture Photo credit: Bei Xiao, Ted Adelson
3
We know how to render them
Monte-Carlo rendering material parameters ? TODO: Maybe replace milk with soap in these examples. rendered image Veach 1997, Dutré et al. 2006
4
We show how to measure them
inverse rendering material parameters TODO: Maybe replace milk with soap in these examples. captured photograph rendered image
5
Our contributions 1. exact inverse volume rendering
material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick 3. database of broad range of materials non-dilutable solids
6
Why is inverse rendering so hard?
random walk of photons inside volume radiative transfer volume light transport has very complex dependence material parameters material sample thin thick non-dilutable solids
7
Light transport approximations
single-bounce random walk random walk of photons inside volume Previous approach: single-scattering Narasimhan et al. 2006 thin thick non-dilutable solids
8
Light transport approximations
isotropic distribution of photons random walk of photons inside volume Previous approach: diffusion Jensen et al. 2001 Papas et al. 2013 … … … … parameter ambiguity material 1 thin thick non-dilutable solids ≈ ≠ material 2
9
Inverse rendering without approximations
exact inversion of random walk random walk of photons inside volume thin thick non-dilutable solids
10
Our approach appearance matching i. material representation
ii. operator-theoretic analysis iii. stochastic optimization
11
random walk of photons inside medium
Background random walk of photons inside medium θ extinction coefficient σt m = (σt σs p(θ)) scattering coefficient σs phase function p(θ)
12
Phase function parameterization
Previous approach: single-parameter families Henyey-Greenstein lobes Chen et al. 2006 Donner et al. 2008 Fuchs et al. 2007 g∈ −1,1 Goesele et al. 2004 Gu et al. 2008 Hawkins et al. 2005 not general enough Holroyd et al. 2011 Gkioulekas et al. 2013 Jensen et al. 2001 McCormick et al. 1981 Narasimhan et al. 2006 Papas et al. 2013 Pine et al. 1990 Prahl et al. 1993 Wang et al. 2008
13
Dictionary parameterization
tent phase functions dictionary of phase functions materials D = {m1, m2, …, mQ} D = {p1, p2, …, pQ} p8 p9 p10 p11 p7 p4 p2 p3 p6 p5 p1 D arbitrary materials phase functions π5 π4 π3 π8 π7 π6 π9 π2 π10 m = Σq πq mq p = Σq πq pq π11 π1 p similarly for σt and σs σt = Σq πq σt,q σs = Σq πq σs,q
14
Our approach appearance matching i. material representation
m = Σq πq mq i. material representation ii. operator-theoretic analysis iii. stochastic optimization
15
Operator-theoretic analysis
random walk of photons inside medium discretized random walk paths propagation step τ τ τ τ τ m = (σt σs p(θ))
16
Operator-theoretic analysis
radiance at all medium points and directions discretized random walk paths propagation step τ Ln+1(x, θ) = Ln(x, θ) K radiance after n+1 steps total radiance radiance after n steps L = Σn Ln = (I - K)-1 Linput rendering operator R L(x, θ) L(x, θ) = R Linput(x, θ) dictionary representation: m = (σt σs p(θ)) m = Σq πq mq K(π) = Σq πq Kq R(π)= (I - Σ q πq Kq)-1
17
Our approach appearance matching i. material representation
m = Σq πq mq i. material representation R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis iii. stochastic optimization
18
Stochastic optimization
appearance matching min ǁ photo - render(π) ǁ2 π analytic operator expression for gradient! 𝜕loss π 𝜕 π q = render(π) single-stepq render(π) R(π) Kq R(π) gradient descent optimization for inverse rendering
19
Stochastic optimization
exact gradient descent N = a few hundreds for k = 1, …, N, πk = πk ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * exact several CPU hours = intractable
20
Stochastic optimization
Monte-Carlo rendering to compute 𝜕loss π 𝜕 π q 𝜋 𝑘−1 102 samples 104 samples 106 samples noisy + fast accurate + slow
21
Stochastic optimization
exact gradient descent N = a few hundreds for k = 1, …, N, πk = πk ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * exact several CPU hours = intractable stochastic gradient descent N = a few hundreds for k = 1, …, N, πk = πk ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * noisy few CPU seconds = solvable
22
Theory wrap-up appearance matching min ǁ photo - render(π) ǁ2
m = Σq πq mq i. material representation R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis noisy 𝜕loss π 𝜕 π q iii. stochastic optimization
23
Our contributions 1. exact inverse volume rendering
material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick 3. database of broad range of materials non-dilutable solids
24
Measurements appearance matching min ǁ photo - render(π) ǁ2
multiple lighting multiple viewpoints
25
Acquisition setup material sample frontlighting camera backlighting
26
Acquisition setup material sample frontlighting material sample
backlighting frontlighting camera camera backlighting top rotation stage top rotation stage bottom rotation stage bottom rotation stage
27
Validation calibration materials Mie theory known parameters
medium material Mie theory particle material size % known parameters polystyrene monodispersions aluminum oxide polydispersions very precise dispersions (NIST Traceable Standards) Frisvad et al. 2007
28
Parameter accuracy comparison of ground-truth and measured parameters
θ -π π p(θ) polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide all parameters estimated within 4% error ground-truth measured Henyey-Greenstein fit
29
Matching novel measurements
comparison of captured and rendered images captured rendered rendered with HG profiles polystyrene 3 images under unseen geometries predicted within 5% RMS error ground-truth measured Henyey-Greenstein fit
30
Our contributions 1. exact inverse volume rendering
material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick non-dilutable solids 3. database of broad range of materials
31
Measured materials thin thick non-dilutable solids hand cream
olive oil curacao shampoo robitussin mixed soap whole milk milk soap wine liquid clay mustard coffee reduced milk thin thick non-dilutable solids
32
Measured phase functions
whole milk reduced milk mustard shampoo hand cream liquid clay milk soap mixed soap glycerine soap robitussin coffee olive oil curacao wine θ -π π p(θ) measured Henyey-Greenstein fit
33
Measured phase functions
whole milk reduced milk mustard shampoo hand cream liquid clay milk soap mixed soap glycerine soap robitussin coffee olive oil curacao wine θ -π π p(θ) measured Henyey-Greenstein fit
34
Synthetic images mixed soap glycerine soap olive oil curacao
whole milk rendered image
35
Synthetic images chromaticity
36
Synthetic images mixed soap glycerine soap olive oil curacao
whole milk rendered image
37
Effect of phase function
measured phase function Henyey-Greenstein fit rendered image chromaticity p(θ) mixed soap θ measured -π π Henyey-Greenstein fit
38
Discussion more interesting materials: more general solids, heterogeneous volumes, fluorescing materials other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging) faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability
39
Take-home messages 1. exact inverse volume rendering
material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick non-dilutable solids 3. database of broad range of materials
40
Acknowledgements Henry Sarkas (Nanophase) Wenzel Jakob (Mitsuba)
Funding: National Science Foundation European Research Council Binational Science Foundation Feinberg Foundation Intel Amazon Database of measured materials:
41
Error surface appearance matching min ǁ photo - render(π) ǁ2 π
42
Light generation MEMS light switch blue (480 nm) laser green (535 nm)
RGB combiner red (635 nm) laser
Similar presentations
