Light field microscopy Marc Levoy, Ren Ng, Andrew Adams Matthew Footer, Mark Horowitz Stanford Computer Graphics Laboratory

 Marc Levoy Executive summary captures the 4D light field inside a microscope yields perspective flyarounds and focal stacks from a single snapshot, but at lower spatial resolution focal stack → deconvolution microscopy → volume data

 Marc Levoy Devices for recording light fields small scenes big scenes handheld camera[Buehler 2001] camera gantry[Stanford 2002] array of cameras[Wilburn 2005] plenoptic camera[Ng 2005] light field microscope(this paper) (using geometrical optics)

 Marc Levoy Light fields at micron scales wave optics must be considered –diffraction limits the spatial × angular resolution most objects are no longer opaque –each pixel is a line integral through the object »of attenuation »or emission –can reconstruct 3D structure from these integrals »tomography »3D deconvolution

 Marc Levoy Conventional versus plenoptic camera

 Marc Levoy Conventional versus plenoptic camera uv-planest-plane square-sided microlenses 125μ

 Marc Levoy Digital refocusing refocusing = summing windows extracted from several microlenses Σ Σ

 Marc Levoy Example of digital refocusing

 Marc Levoy Refocusing portraits

 Marc Levoy Macrophotography

 Marc Levoy Digitally moving the observer moving the observer = moving the window we extract from the microlenses Σ Σ

 Marc Levoy Example of moving the observer

 Marc Levoy Moving backward and forward

 Marc Levoy A light field microscope (LFM) objective specimen intermediate image plane eyepiece

 Marc Levoy A light field microscope (LFM) 40x / 0.95NA objective ↓ 0.26μ spot on specimen × 40x = 10.4μ on sensor ↓ 2400 spots over 25mm field micron microlenses ↓ 200 × 200 microlenses with 12 × 12 spots per microlens objective specimen intermediate image plane eyepiece sensor → reduced lateral resolution on specimen = 0.26μ × 12 spots = 3.1μ

 Marc Levoy A light field microscope (LFM) sensor 160mm 2.5mm 0.2mm

 Marc Levoy Example light field micrograph orange fluorescent crayon mercury-arc source + blue dichroic filter 16x / 0.5NA (dry) objective f/20 microlens array 65mm f/2.8 macro lens at 1:1 Canon 20D digital camera ordinary microscope light field microscope 200μ

 Marc Levoy The geometry of the light field in a microscope microscopes make orthographic views translating the stage in X or Y provides no parallax on the specimen out-of-plane features don’t shift position when they come into focus f objective lenses are telecentric

 Marc Levoy Panning and focusing panning sequencefocal stack

 Marc Levoy Mouse embryo lung (16x / 0.5NA water immersion) light field panfocal stack 200μ

 Marc Levoy Axial resolution (a.k.a. depth of field) wave term + geometrical optics term ordinary microscope (16x/0.4NA (dry), e = 0) with microlens array (e = 125μ) stopped down to one pixel per microlens → number of slices in focal stack = 12 (geometrical optics dominates) (wave optics dominates)

 Marc Levoy 3D reconstruction confocal scanning[Minsky 1957] shape-from-focus[Nayar 1990] deconvolution microscopy[Agard 1984] –4D light field → digital refocusing → 3D focal stack → deconvolution microscopy → 3D volume data (UMIC SUNY/Stonybrook) (Noguchi) (DeltaVision)

 Marc Levoy 3D deconvolution object * PSF → focus stack  {object} ×  {PSF} →  {focus stack}  {focus stack}   {PSF} →  {object} spectrum contains zeros, due to missing rays imaging noise is amplified by division by ~zeros reduce by regularization, e.g. smoothing focus stack of a point in 3-space is the 3D PSF of that imaging system [McNally 1999]  {PSF}

 Marc Levoy Silkworm mouth (40x / 1.3NA oil immersion ) slice of focal stackslice of volumevolume rendering 100μ

 Marc Levoy Insect legs (16x / 0.4NA dry ) volume rendering all-focus image [Agarwala 2004] 200μ

 Marc Levoy 3D reconstruction (revisited) 4D light field → digital refocusing → 3D focal stack → deconvolution microscopy → 3D volume data 4D light field → tomographic reconstruction → 3D volume data (from Kak & Slaney) (DeltaVision)

 Marc Levoy Implications of this equivalence light fields of minimally scattering volumes contain only 3D worth of information, not 4D the extra dimension serves to reduce noise, but could be re-purposed? Optical Projection Tomography [Sharpe 2002]

 Marc Levoy Conclusions captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time Calcium fluorescent imaging of zebrafish larvae optic tectum during changing visual stimula

 Marc Levoy Conclusions captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time but... sacrifices spatial resolution to obtain control over viewpoint and focus 3D reconstruction fails if specimen is too thick or too opaque

 Marc Levoy Future work extending the field of view by correcting digitally for objective aberrations Nikon 40x 0.95NA (dry) Plan-Apo

 Marc Levoy Future work extending the field of view by correcting digitally for objective aberrations microlenses in the illumination path → an imaging microscope scatterometer 200μ angular dependence of reflection from single squid iridophore

 Marc Levoy