Lensless Imaging Richard Baraniuk Rice University Ashok Veeraraghavan Aswin Sankaranarayanan CMU John Rogers UIUC Richard Baraniuk Rice University

Re-Imagining Imaging Conventional camera design based on human visual system model objective lens: directs a cone of light into the camera 1-to-1 correspondence between scene points and camera pixels makes imaging easy Inspired by the discovery of light sensing opsin in the skin of cephalopods, we are developing new kind of camera architectures based on distributed light sensing. Such new cameras promise exciting new form factors (eg: flat, flexible)

Re-Imagining Imaging Conventional camera design based on human visual system model objective lens: directs a cone of light into the camera 1-to-1 correspondence between scene points and camera pixels makes imaging easy Goal: A large, potentially flexible, imaging platform capable of distributed acquisition of light fields inspired by distributed light sensing in cephalopod skin Approach: Lensless imaging leverage recent progress in coded aperture and compressive sensing exciting opportunities for flat and flexible cameras Inspired by the discovery of light sensing opsin in the skin of cephalopods, we are developing new kind of camera architectures based on distributed light sensing. Such new cameras promise exciting new form factors (eg: flat, flexible)

Problem With incoherent light, no phase information all photo-detectors measure roughly the same information, the average light level of the scene sensor scene photo-detector 1 photo-detector 2

Solution Add a mask in front of the sensor/photo-detector attenuates certain rays of light How does this help? same scene point is attenuated differently at different photo-detectors each photo-detector sees a different linear combination of the scene points can design mask(s) such that we can recover a high-resolution version of the scene (compressive sensing) mask sensor scene photo-detector 1 photo-detector 2

Mask Design Random mask rich theory and algorithms available from compressive sensing provable recovery bounds sparse signal measurements nonzero entries super sub-Nyquist measurement

Mask Design Random mask rich theory and algorithms available from compressive sensing provable recovery bounds major impact in a variety of DOD and industrial sensing systems: medical, radar, sonar, hyperspectral, IR, THz imaging, …

f( ) = X Simulations 10 units 1/10 unit mask(s) sensor scene X sensor measurements mask = f( ) X image noiseless system PSNR = 19dB noisy system PSNR = 15dB

Planar Prototype Sensor: Flea3 Point Grey camera, 1024x1280 pixels (5.3µm each) Mask: Random binary mask (1) 135x135 features (85µm each) 10% of mask “pixels” transparent Target projected on screen 15cm from camera (target height/width 24cm) sensor-mask assembly walls to limit FOV mask target on monitor gap (0.5mm) mount sensor sensor image

Planar Prototype Results 1 target on screen recovered image Sensor: Flea3 Point Grey camera, 1024x1280 pixels (5.3µm each) Mask: Random binary mask (1) 135x135 features (85µm each) 10% of mask “pixels” transparent Target projected on screen 15cm from camera (target height/width 24cm) Reconstruction: Least-squares Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

Planar Prototype Results 2 target on screen recovered image Sensor: Flea3 Point Grey camera, 1024x1280 pixels (5.3µm each) Mask: Random binary mask (1) 270x270 features (42µm each) 10% of mask “pixels” transparent Target projected on screen 15cm from camera (target height/width 24cm) Reconstruction: Least-squares Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

Color Images

From Challenge … With a planar sensor, must limit the field of view Outside field of view, image recovery becomes increasingly ill-posed scene X sensor mask(s)

… To Opportunity Without the need for a lens, we can make the mask and sensor curved Example: (Hemi)spherical camera 180/360 degree field of view no field-of-view ill-posedness! mask(s) sensor array with John Rogers

… To Opportunity Without the need for a lens, we can make the mask and sensor curved Example: (Hemi)spherical camera 180/360 degree field of view no field-of-view ill-posedness! spherical sensor array scene projected on sphere un-warped recovered image

Simulations additive noise noiseless scene 𝜎=0.1 𝜎=0.5 𝜎=1 512 256 256 256 128 128 128 512 Number of pixels in reconstruction

Spherical Prototype Sensor: White paint on a spherical shell acts as diffuser Mask: random binary mask 68x68 features (170µm each) 10% of mask “pixels” transparent Target projected on screen 18cm away (target height/width 24cm) plastic shell with diffuser inner surface (proxy for a spherical sensor) current prototype uses a Grasshopper point grey camera to capture image formed on a 1.2 cm2 area (400x400 pixels) planar mask in front of shell (flexible PDMS masks on shell)

Spherical Prototype Results Sensor: White paint on a spherical shell acts as diffuser Mask: random binary mask 68x68 features (170µm each) 10% of mask “pixels” transparent Target projected on screen 18cm away (target height/width 24cm) Reconstruction: Least-squares Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure Spherical photo-detector to improve SNR targets on screen recovered images 32x32 recovered images

Planned Research flat, flexible, (hemi)spherical cameras Radically new kinds of cameras flat, flexible, (hemi)spherical cameras beyond visible (IR, THz, …) numerous potential DOD and industrial applications New theory and algorithms mask design (light throughput versus invertibility) dynamic masks new recovery algorithms needed for 0/1 masks Can perform exploitation directly on compressive measurements (detection/classification, etc.) without numerical scene reconstruction Sensing light fields instead of images