Linear View Synthesis Using a Dimensionality Gap Light Field Prior Anat Levin and Fredo Durand Weizmann Institute of Science & MIT CSAIL 1

Light fields 2 Light field: the set of rays emitted from a scene in all possible directions

(Animation by Marc Levoy) Light fields 3 Novel view rendering (Animation by Marc Levoy)

(Animation by Marc Levoy) Light fields 4 Novel view rendering (Animation by Marc Levoy)

(Animation by Marc Levoy) Light fields 5 Novel view rendering Synthetic refocusing (Animation by Marc Levoy)

4D light field 6 u v The set of light rays hitting the camera aperture plane is 4D: Ray hitting point- 2D Ray orientation- 2D (In general: a 7D plenoptic space, including time and wavelength dimensions)

Light field acquisition schemes and priors 7 Very different approaches to light field acquisition and manipulations exist in the literature. The inherent difference between them is a different prior model on the light field space

Light field acquisition schemes and priors 8 4D: The light field is smooth, but involves 4 degrees of freedom -Capture: 4D data (e.g. camera array) -Inference: linear .

Light field acquisition schemes and priors 9 4D: -Capture: 4D data (e.g. camera array) -Inference: linear 2D: For Lambertian scenes all rays emerging from one point have same color. If depth is known, only 2 degrees of freedom -Capture: 2D data (e.g. stereo camera) -Inference: non linear depth estimation

In this talk: 3D light field prior 10 4D: -Capture: 4D data (e.g. camera array -Inference: linear 2D: -Capture: 2D data (e.g. stereo camera) -Inference: non linear depth estimation 3D: Depth is a 1D variable, hence the union of images at any depth covers no more than a 3D subset. Show that in the frequency domain there is only a 3D manifold of non zero entries. -Capture: 3D data (e.g. focal stack)

Outline Linear view synthesis from a focal stack sequence 11 Linear view synthesis from a focal stack sequence The 3D light field prior Frequency derivation of synthesis algorithm Other applications of the 3D prior

Linear view synthesis with 3D prior 12 Input: Focal stack (3D data) 1D set of 2D images focused at different depth Output: Novel viewpoints (4D data) 2D Images x 2D set of novel viewpoints Linear image processing

Linear view synthesis algorithm 13 No depth estimation! Shift focal stack images by disparity of desired view Average shifted images Depth invariant deconvolution 1 2 3

Shift invariant convolution~ focus sweep camera 14 Average shifted images Depth invariant blur kernel Ideal pinhole image Inspiration: The focus sweep camera Hausler 72, Nagahara et al. 08 Captures a single image, average over all focus depths during exposure, provides EDOF image from a single view

Linear view synthesis results 15 Video animation here

Disclaimers Novel viewpoints limited to the aperture area 16 Novel viewpoints limited to the aperture area Convolution model breaks at occlusion boundaries Assume scene is Lambertian- in practice holds within the narrow range of angles of the aperture

Outline Linear view synthesis from a focal stack sequence 17 Linear view synthesis from a focal stack sequence The 3D light field prior Frequency derivation of synthesis algorithm Other applications of the 3D prior

4D light field v y v x u u (x,y,u,v) 18 u v u v y x y x (x,y,u,v) The set of light rays hitting the lens is 4D

4D light field v y v x u u (?,?,u0,0) (x,y,u,v) 19 u v u v y x y x (?,?,u0,0) (x,y,u,v) The set of light rays hitting the lens is 4D

4D light field v y v x u u (?,?,0,v0) (x,y,u,v) 20 u v u v y x y x (?,?,0,v0) (x,y,u,v) The set of light rays hitting the lens is 4D

4D light field v y v x u u (x,y,u,v) 21 u v u v y x y x (x,y,u,v) The set of light rays hitting the lens is 4D

4D light field spectrum y v x u (x,y,u,v) L( , , , ) 22 v y x u 4D Fourier Transform (x,y,u,v) The set of light rays hitting the lens is 4D Study the 4D Fourier domain L( , , , )

4D light field spectrum y v x u L( ,0,?,?) (x,y,u,v) L( , , , ) 23 v y x u 4D Fourier Transform L( ,0,?,?) (x,y,u,v) The set of light rays hitting the lens is 4D Study the 4D Fourier domain L( , , , )

4D light field spectrum y v x u 24 v y x u 4D Fourier Transform Frequency content only along 1D segments

Energy portion away from focal segments 4D light field spectrum Scene 4D Light field spectrum Energy portion away from focal segments

The slicing theorem y v x u 4D Fourier Transform 26 v y x u 4D Fourier Transform 2D focused images at varying depths 2D Fourier Transform

The dimensionality gap 27 v y x u far 4D Fourier Transform depth color coding near Light field spectrum: 4D Image spectrum: 2D Depth: 1D → Dimensionality gap (Ng 05, Levin et al. 09) Only the 3D manifold corresponding to physical focusing distance is useful 3D

3D Gaussian light field prior 28 Gaussian prior: assigns non zero variance only to 3D set of entries on the focal segments Gaussian=> inference simple and linear Focal stack directly samples the manifold with non zero variance

Outline Linear view synthesis from a focal stack sequence 29 Linear view synthesis from a focal stack sequence The 3D light field prior Frequency derivation of synthesis algorithm Other applications of the 3D prior

View synthesis in the frequency domain 30 4D spectrum of constant depth scene Average focal stack spectra Spectra of correct depth Sample density Spectra of focal stack images Deconvolution (frequency domain)

Outline Linear view synthesis from a focal stack sequence 31 Linear view synthesis from a focal stack sequence The 3D light field prior Frequency derivation of synthesis algorithm Other applications of the 3D prior

Prior to infer light field from partial samples 32 In many other light field acquisition schemes we capture only a partial information on the light field- limited resolution, aliasing and each. However, we capture linear measurements On the other hand, we have a Gaussian prior, and we know the light field actually occupies only a low dimensional manifold of the 4D space. Use the prior to “invert the rank deficient projection” and interpolate the measurements to get a light field with higher resolution, less aliasing.

Improved viewpoints sample 33 4D Light field acquisition systems sample a 2D set of view points Can we do with sparser sample and 3D Gaussian prior for interpolation? How many samples needed? What is the right spacing? Shall we distribute samples on a grid? Better arrangement? Grid: Standard sampling pattern Circle: Sampling pattern with improved reconstruction using 3D prior

Superesolution of plenoptic camera measurements 34 Plenoptic camera measurements are aliased Replicas off the focal segments are high frequencies which we can re-bin and restore high frequency information

Superesolution of plenoptic camera measurements 35 Bicubic interpolation Lumsdaine and Georgiev: applies for a single known depth Our result: applies for all depths simultaneously, no depth estimation

Summary 36 Light field acquisition and synthesis strongly depends on light field prior Existing priors: Linear view synthesis from the focal stack Other applications of 3D prior: - viewpoints sample pattern - depth invariant superesolution of plenoptic camera data 4D prior: capture- 4D data (e.g. camera array), inference- linear 2D prior: capture- 2D data (e.g. stereo), inference- non linear Our new prior: 3D prior: capture- 3D data (e.g. focal stuck), inference linear