Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas1 Shuang Zhao2 Kavita Bala2 Todd Zickler1 Anat Levin3 1Harvard 2Cornell 3Weizmann

Most materials are translucent food skin jewelry architecture Photo credit: Bei Xiao, Ted Adelson

We know how to render them Monte-Carlo rendering material parameters ? TODO: Maybe replace milk with soap in these examples. rendered image Veach 1997, Dutré et al. 2006

We show how to measure them inverse rendering material parameters TODO: Maybe replace milk with soap in these examples. captured photograph rendered image

Our contributions 1. exact inverse volume rendering material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick 3. database of broad range of materials non-dilutable solids

Why is inverse rendering so hard? random walk of photons inside volume radiative transfer volume light transport has very complex dependence material parameters material sample thin thick non-dilutable solids

Light transport approximations single-bounce random walk random walk of photons inside volume Previous approach: single-scattering Narasimhan et al. 2006   thin thick non-dilutable solids  

Light transport approximations isotropic distribution of photons random walk of photons inside volume Previous approach: diffusion Jensen et al. 2001 Papas et al. 2013 … … … … parameter ambiguity   material 1 thin thick non-dilutable solids ≈ ≠   material 2

Inverse rendering without approximations exact inversion of random walk random walk of photons inside volume   thin thick non-dilutable solids  

Our approach appearance matching i. material representation ii. operator-theoretic analysis iii. stochastic optimization

random walk of photons inside medium Background random walk of photons inside medium θ extinction coefficient σt m = (σt σs p(θ)) scattering coefficient σs phase function p(θ)

Phase function parameterization Previous approach: single-parameter families Henyey-Greenstein lobes Chen et al. 2006 Donner et al. 2008 Fuchs et al. 2007 g∈ −1,1 Goesele et al. 2004 Gu et al. 2008 Hawkins et al. 2005 not general enough Holroyd et al. 2011 Gkioulekas et al. 2013 Jensen et al. 2001 McCormick et al. 1981 Narasimhan et al. 2006 Papas et al. 2013 Pine et al. 1990 Prahl et al. 1993 Wang et al. 2008

Dictionary parameterization tent phase functions dictionary of phase functions materials D = {m1, m2, …, mQ} D = {p1, p2, …, pQ} p8 p9 p10 p11 p7 p4 p2 p3 p6 p5 p1 D arbitrary materials phase functions π5 π4 π3 π8 π7 π6 π9 π2 π10 m = Σq πq mq p = Σq πq pq π11 π1 p similarly for σt and σs σt = Σq πq σt,q σs = Σq πq σs,q

Our approach appearance matching i. material representation m = Σq πq mq i. material representation ii. operator-theoretic analysis iii. stochastic optimization

Operator-theoretic analysis random walk of photons inside medium discretized random walk paths propagation step τ τ τ τ τ m = (σt σs p(θ))

Operator-theoretic analysis radiance at all medium points and directions discretized random walk paths propagation step τ Ln+1(x, θ) = Ln(x, θ) K radiance after n+1 steps total radiance radiance after n steps L = Σn Ln = (I - K)-1 Linput rendering operator R L(x, θ) L(x, θ) = R Linput(x, θ) dictionary representation: m = (σt σs p(θ)) m = Σq πq mq K(π) = Σq πq Kq R(π)= (I - Σ q πq Kq)-1

Our approach appearance matching i. material representation m = Σq πq mq i. material representation R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis iii. stochastic optimization

Stochastic optimization appearance matching min ǁ photo - render(π) ǁ2 π analytic operator expression for gradient! 𝜕loss π 𝜕 π q = render(π) · single-stepq · render(π) R(π) Kq R(π) gradient descent optimization for inverse rendering

Stochastic optimization exact gradient descent N = a few hundreds for k = 1, …, N, πk = πk - 1 - ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * exact several CPU hours = intractable

Stochastic optimization Monte-Carlo rendering to compute 𝜕loss π 𝜕 π q 𝜋 𝑘−1 102 samples 104 samples 106 samples noisy + fast accurate + slow

Stochastic optimization exact gradient descent N = a few hundreds for k = 1, …, N, πk = πk - 1 - ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * exact several CPU hours = intractable stochastic gradient descent N = a few hundreds for k = 1, …, N, πk = πk - 1 - ak 𝜕loss π 𝜕 π q 𝜋 𝑘−1 end * noisy few CPU seconds = solvable

Theory wrap-up appearance matching min ǁ photo - render(π) ǁ2 m = Σq πq mq i. material representation R(π)= (I - Σ q πq Kq)-1 ii. operator-theoretic analysis noisy 𝜕loss π 𝜕 π q iii. stochastic optimization

Our contributions 1. exact inverse volume rendering material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick 3. database of broad range of materials non-dilutable solids

Measurements appearance matching min ǁ photo - render(π) ǁ2 multiple lighting multiple viewpoints

Acquisition setup material sample frontlighting camera backlighting

Acquisition setup material sample frontlighting material sample backlighting frontlighting camera camera backlighting top rotation stage top rotation stage bottom rotation stage bottom rotation stage

Validation calibration materials Mie theory known parameters medium material Mie theory particle material size % known parameters polystyrene monodispersions aluminum oxide polydispersions very precise dispersions (NIST Traceable Standards) Frisvad et al. 2007

Parameter accuracy comparison of ground-truth and measured parameters θ -π π p(θ) polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide all parameters estimated within 4% error ground-truth measured Henyey-Greenstein fit

Matching novel measurements comparison of captured and rendered images captured rendered rendered with HG profiles polystyrene 3 images under unseen geometries predicted within 5% RMS error ground-truth measured Henyey-Greenstein fit

Our contributions 1. exact inverse volume rendering material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick non-dilutable solids 3. database of broad range of materials

Measured materials thin thick non-dilutable solids hand cream olive oil curacao shampoo robitussin mixed soap whole milk milk soap wine liquid clay mustard coffee reduced milk thin thick non-dilutable solids

Measured phase functions whole milk reduced milk mustard shampoo hand cream liquid clay milk soap mixed soap glycerine soap robitussin coffee olive oil curacao wine θ -π π p(θ) measured Henyey-Greenstein fit

Measured phase functions whole milk reduced milk mustard shampoo hand cream liquid clay milk soap mixed soap glycerine soap robitussin coffee olive oil curacao wine θ -π π p(θ) measured Henyey-Greenstein fit

Synthetic images mixed soap glycerine soap olive oil curacao whole milk rendered image

Synthetic images chromaticity

Synthetic images mixed soap glycerine soap olive oil curacao whole milk rendered image

Effect of phase function measured phase function Henyey-Greenstein fit rendered image chromaticity p(θ) mixed soap θ measured -π π Henyey-Greenstein fit

Discussion more interesting materials: more general solids, heterogeneous volumes, fluorescing materials other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging) faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability

Take-home messages 1. exact inverse volume rendering material 1. exact inverse volume rendering with arbitrary phase functions! known parameters 2. validation with calibration materials thin thick non-dilutable solids 3. database of broad range of materials

Acknowledgements Henry Sarkas (Nanophase) Wenzel Jakob (Mitsuba) Funding: National Science Foundation European Research Council Binational Science Foundation Feinberg Foundation Intel Amazon Database of measured materials: http://tinyurl.com/sa2013-inverse

Error surface appearance matching min ǁ photo - render(π) ǁ2 π

Light generation MEMS light switch blue (480 nm) laser green (535 nm) RGB combiner red (635 nm) laser