1. Imaging Through Multiple Scattering Media Using Phase Retrieval
Chris Metzler
Phil Schniter, Manoj Sharma, Sudarshan Nagesh, Oliver Cossairt, Ashok Veeraraghavan, Richard Baraniuk
Original Title: An Expectation-Maximization Approach to Tuning Generalized Vector Approximate Message Passing 
20 min including Q&A

2. Multiple Scattering Media
Photon paths
A multiple scattering medium is simply a material that, when light passes through it, causes the photons to undergo multiple reflections.
Multiple Scatterer

3. Multiple Scattering Media are Commonplace
Fog
Snow
Water
Matt Surfaces
(Reflections)
Tissue
Frosted Glass
Multiple scattering media are commonplace and show up in a variety of applications. Including navigation; where we’d might like to see through fog, snow, or water. Medical imaging, where we’d like to see through tissue using visible light. And surveillance, where we might like to use the light scattered through a translucent material or scattered off a matt surface.

4. Phase Retrieval to Image Through Scattering Media
Image Through Thick and Static Scatterer – Frosted Glass
[Angelique Dremea et al. 2015]
Image Through (Simulated) Thin and Dynamic Scatterer – Tissue
[Ori Katz et al. 2014]

5. This Talk Image Through Thick and Static Scatterer – Frosted Glass
[Angelique Dremea et al. 2015]
Image Through (Simulated) Thin and Dynamic Scatterer – Tissue
[Ori Katz et al. 2014]

6. Transmission Matrices (TM) Characterize Scattering Media
Scatterer
This work is about characterizing multiple scattering materials using transmission matrices
Scatterer can be volumetric

7. Transmission Matrices (TM) Characterize Scattering Media
Scatterer
Camera

8. A Simple Experiment Spatial Light Modulator (SLM)
Lets us control the incident wavefront x
Amplitude-SLM
Scatterer
Coherent
Illumination
As an illustration of the ideas behind transmission matrices, we set up he following simple experiment.
SLM is changing the angle of the light incident on the scatterer

9. Amplitude-SLM Pattern
A Simple Experiment
Amplitude-SLM Pattern
Scatterer
Camera
Coherent
Illumination
As an illustration of the ideas behind transmission matrices, we set up he following simple experiment.
SLM is changing the angle of the light incident on the scatterer
Measurements
Magnitudes of columns of the transmission matrix

10. Another Simple Experiment
Ampltidue-SLM Pattern
Scatterer
Camera
Coherent
Illumination
Measurements
As another simple experiment, we set the SLM pattern to be a smiley face. And you can see that when this signal passes through the scattering material constructive and destructive interference causes it to produce a speckler pattern. Also note, that because this scatterer is thick, the speckle pattern bears no resemblance to the SLM pattern
These measurements are called speckle patterns. With thick scatterers they bear no resemblance to the incident wavefront

11. TMs Can Be Used to… Focus Light Through a Scatterer
Phase-SLM Pattern
Scatterer
Foci
Coherent
Illumination
[Popoff et al. 2010]

12. TMs Can Be Used to… Transmit a Signal
Sequence of Phase-SLM Patterns
Scatterer
Foci
Coherent
Illumination
[N’Gom et al. 2017]

13. TMs Can Be Used to… Image Through a Scatterer
Phase SLM with 2 pixels=1 and the rest=-1
Scatterer
Coherent
Illumination
Complex Field Measurements
Reconstruction
Computational processing
[Popoff et al. 2010, Choi et al. 2012]

14. TMs Can Be Used to… Image Past the Diffraction Limit
Resolution
Target
Scatterer
Coherent
Illumination
Complex Field Measurements
Computational processing
Reconstruction
Scatterer increases the Aperture
[Choi et al. 2012]

15. TMs Can Be Used to… Image Past the Diffraction Limit
All these applications require first measuring the TM.
This work focuses on learning TMs.
Resolution
Target
Scatterer
Coherent
Illumination
Complex Field Measurements
Computational processing
Reconstruction
[Choi et al. 2012]

16. Measuring a TM: Holographic Interferometry [Popoff et al. 2010]
Beam Splitter
Scatterer
SLM
Camera
Coherent
Illumination
Interference Pattern
Use interferometry to capture complex-valued measurements
Determine complex-valued TM with complex-valued measurements
Experimentally difficult
Computationally trivial

17. Measuring a TM: Holographic Interferometry
Illumination Patterns
Beam Splitter
Scatterer
SLM
Complex Measurements
Step 1: Capture a series of complex-valued measurements

18. Measuring a TM: Holographic Interferometry
Illumination Patterns
Beam Splitter
Scatterer
SLM
Complex Measurements
Step 1: Capture a series of complex-valued measurements
Step 2: Stack together and then transpose the measurements

19. Measuring a TM: Holographic Interferometry
Illumination Patterns
Beam Splitter
Scatterer
SLM
Complex Measurements
Step 1: Capture a series of complex-valued measurements
Step 2: Stack together and then transpose the measurements
Step 3: Learn TM by solving a linear inverse problem

20. Measuring a TM: Holographic Interferometry
Illumination Patterns
Beam Splitter
Scatterer
SLM
Complex Measurements
Step 1: Capture a series of complex-valued measurements
Step 2: Stack together and then transpose the measurements
Step 3: Learn TM by solving a linear inverse problem
Interferometric measurements (complex-valued) are extremely sensitive to vibrations and outside interference
Interferometry was recently used to measure gravitational waves: m sensitivity! [Martynov et al. 2016]

21. Measuring a TM: Double Phase Retrieval [Drémeau et al
Measuring a TM: Double Phase Retrieval [Drémeau et al. 2015, Rajaei et al. 2016]
Use camera to capture real-valued measurements
Determine complex-valued TM with real-valued measurements
Experimentally easy
No interferometry
Can use digital micromirror device (DMD) as SLM; very fast
Computationally difficult
Scatterer
SLM
Camera
Coherent
Illumination

22. Measuring a TM: Double Phase Retrieval [Drémeau et al
Measuring a TM: Double Phase Retrieval [Drémeau et al. 2015, Rajaei et al. 2016]
Illumination Patterns
Magnitude Measurements
Step 1: Capture a series of calibration images
Scatterer
SLM
Camera
Coherent
Illumination
Drémeau came up with the idea of

23. Measuring a TM: Double Phase Retrieval [Drémeau et al
Measuring a TM: Double Phase Retrieval [Drémeau et al. 2015, Rajaei et al. 2016]
Illumination Patterns
Magnitude Measurements
Step 1: Capture a series of calibration images
Step 2: Stack together and then transpose the measurements
Scatterer
SLM
Camera
Coherent
Illumination
Drémeau came up with the idea of

24. Measuring a TM: Double Phase Retrieval [Drémeau et al
Measuring a TM: Double Phase Retrieval [Drémeau et al. 2015, Rajaei et al. 2016]
Illumination Patterns
Magnitude Measurements
Step 1: Capture a series of calibration images
Step 2: Stack together and then transpose the measurements
Step 3: Learn TM by solving a series of phase retrieval problems
Scatterer
SLM
Camera
Coherent
Illumination
Drémeau came up with the idea of
Solve for one column of AH at a time

25. Camera resolution determines number of phase retrieval problems.
Measuring a TM: Double Phase Retrieval [Drémeau et al. 2015, Rajaei et al. 2016]
Illumination Patterns
Magnitude Measurements
Step 1: Capture a series of calibration images
Step 2: Stack together and then transpose the measurements
Step 3: Learn TM by solving a series of phase retrieval problems
Scatterer
SLM
Camera
Coherent
Illumination
Target resolution determines the dimension of phase retrieval problems.
Camera resolution determines number of phase retrieval problems.
256x256 camera resolution and 64x64 target resolution => 200,000 CPU hours with existing methods!
Drémeau came up with the idea of
Solve for one column of AH at a time

26. Why are computation times so bad?
Most phase retrieval algorithms worth with Gaussian measurements
PhasePack
Tom U Maryland
How data is used now and what’s wrong with that
Show gaussian vs binary measurement matrix reconstructions
Provide computation times

27. Why are computation times so bad?
Very few phase retrieval algorithms work well with {0,1} measurement matrices
How data is used now and what’s wrong with that
Show gaussian vs binary measurement matrix reconstructions
Provide computation times

28. Why are computation times so bad?
prSAMP works…
Note: Results are with multiple restarts (non-convex problem)
How data is used now and what’s wrong with that
Show gaussian vs binary measurement matrix reconstructions
Provide computation times
… but has O(N3) computational complexity; a single N=642 PR problem takes 1 hour

29. Our Solution: A New Algorithm, prVAMP
Extends GVAMP [Schniter et al. 2016] to PR
Approximates solution to
Infuses ADMM concepts into AMP
Updates the entire estimate of x in parallel
Extends class of measurement matrices A to anything “right-rotationally invariant”
Works with binary 0/1 and Gaussian measurements
Does not work with Fourier measurements
Complexity: O(N2) + one-time SVD (and vector operations can be parallelized on GPUs)
Accurate, robust, and fast
right-rotationally invariant: A matrix A is right-rotationally invariant if it can be written in the
form A = USV^t, with V uniformly distributed over the group of orthogonal
matrices.
This new alg is setup to solve problem with what kind of measurements and what kind of structure in x?
Measurements: right-rotationally invariant
Structure in x: theory is for i.i.d prior. Can be extended to handle “denoiser” priors like D-AMP. Currently tested with i.i.d. Gaussian prior.

30. prVAMP Works as Well as PrSAMP
Note: Results are with multiple restarts (non-convex problem)
SNR=20 dB. Plot shows results with multiple restarts.

31. prVAMP is Much Faster
prVAMP is ~100x faster than prSAMP on the same CPU (0/1 measurements and low SNR)
prVAMP easily maps onto a GPU for another 500x speed up
Unfair comparison below: prVAMP is running on one $1k GPU while prSAMP is running on $1M worth of CPUs
prSAMP
prVAMP
prSAMP
prVAMP
These results are for very low SNRs (3dB) and {0,1} measurements (simulations)
Results of left two plots are for 12*32^2x32^2 resolution.
Results on the right are for 12*64^2x64^2 resolution
Note that figure on right does no show a 10000:1 speed up. 10,000:1 number comes from comparing one CPU to one GPU. We are comparing about half a million dollars worth of CPUs to one $1000 GPU
w/ GPU
CPU: 8 core Xeon 2650
GPU: 3584 core Titan X

32. Experimental Validation: Learning a TM
Solve for one column of AH at a time
Illumination Patterns
Measurements
3 GPU Hours: prVAMP
CPU Hours: prSAMP
Diffuser acts as multiple scattering material
Using prVAMP, we learned a 2562x642 TM in
3 GPU hours,
instead of
200,000 CPU hours!

33. Experimental Validation: Imaging with a TM
Measurement

34. Experimental Validation: Imaging with a TM
Reconstruction
Measurement
Phase Retrieval using Â

35. Experimental Validation: Imaging with a TM
Reconstruction
Measurement
Phase Retrieval using Â
Hidden Message

36. Results
Capturing and processing lots of data enables learning largest ever TM
Measurement
256x256 measurements
Measurement
Calibration requires solving PR problems of size 642 (3 hours on one TitanX GPU)
Be clear that the 3hrs is for learning the TM. Reconstructing a single image is a few seconds.
Transmission matrix was learned over 3 hours using a GPU and prVAMP
Double phase retrieval.
PR #1 Calibration:
We solve as many phase retrieval problems as our camera resolution. In our case this is 256^2. Each of these problems reconstructs a row of the TM. Each row is the size of the SLM resolution. Our SLM resolution is 64^2. Each of these problems has a measurement vector is determined by the number of measurements we capture, which is 12x the SLM resolution.
To summarize, during calibration we solve 256^2 independent phase retrieval problems. Each of these phase retrieval problems reconstructs as 64^2-dim vector from a 12*64^2-dim measurement.
PR #2 Imaging:
During imaging, we recostruct the signal on the SLM from the measurements on the camera. In our case, we reconstruct a 64^2 dim vector from a 256^2-dim measurement. For these PR problems, one could use a prior because we know the SLM pattern is real-valued. We do not use a prior because we with natural scenes the signal would be complex.
Measurement

37. Results
Capturing and processing lots of data enables learning largest ever TM
Measurement
Phase Retrieval using Â
Reconstruction
256x256 measurements
Measurement
Phase Retrieval using Â
Reconstruction
Recover 64x64 image on amplitude SLM (few seconds)
Calibration requires solving PR problems of size 642 (3 hours on one TitanX GPU)
Be clear that the 3hrs is for learning the TM. Reconstructing a single image is a few seconds.
Transmission matrix was learned over 3 hours using a GPU and prVAMP
Double phase retrieval.
PR #1 Calibration:
We solve as many phase retrieval problems as our camera resolution. In our case this is 256^2. Each of these problems reconstructs a row of the TM. Each row is the size of the SLM resolution. Our SLM resolution is 64^2. Each of these problems has a measurement vector is determined by the number of measurements we capture, which is 12x the SLM resolution.
To summarize, during calibration we solve 256^2 independent phase retrieval problems. Each of these phase retrieval problems reconstructs as 64^2-dim vector from a 12*64^2-dim measurement.
PR #2 Imaging:
During imaging, we recostruct the signal on the SLM from the measurements on the camera. In our case, we reconstruct a 64^2 dim vector from a 256^2-dim measurement. For these PR problems, one could use a prior because we know the SLM pattern is real-valued. We do not use a prior because we with natural scenes the signal would be complex.
Reconstruction
Measurement
Phase Retrieval using Â

38. Results
Capturing and processing lots of data enables learning largest ever TM
Measurement
Phase Retrieval using Â
Reconstruction
Hidden Message
256x256 measurements
Measurement
Phase Retrieval using Â
Reconstruction
Hidden Message
Recover 64x64 image on amplitude SLM (few seconds)
Calibration requires solving PR problems of size 642 (3 hours on one TitanX GPU)
Be clear that the 3hrs is for learning the TM. Reconstructing a single image is a few seconds.
Transmission matrix was learned over 3 hours using a GPU and prVAMP
Double phase retrieval.
PR #1 Calibration:
We solve as many phase retrieval problems as our camera resolution. In our case this is 256^2. Each of these problems reconstructs a row of the TM. Each row is the size of the SLM resolution. Our SLM resolution is 64^2. Each of these problems has a measurement vector is determined by the number of measurements we capture, which is 12x the SLM resolution.
To summarize, during calibration we solve 256^2 independent phase retrieval problems. Each of these phase retrieval problems reconstructs as 64^2-dim vector from a 12*64^2-dim measurement.
PR #2 Imaging:
During imaging, we recostruct the signal on the SLM from the measurements on the camera. In our case, we reconstruct a 64^2 dim vector from a 256^2-dim measurement. For these PR problems, one could use a prior because we know the SLM pattern is real-valued. We do not use a prior because we with natural scenes the signal would be complex.
Reconstruction
Measurement
Phase Retrieval using Â
Hidden Message

39. A Public Dataset: Test your new PR algorithm with real data
Measurement matrix is ~i.i.d. subguassian
[Goldstein and Studer 2018]

40. Summary: PR Algorithms Can Image Through Scatterers
TMs for thick and static scattering media
Characterize a scatterer by solving tens thousands of PR problems
“Invert” scattering by solving PR problem
This work: Develop fast phase retrieval algorithm - prVAMP
Release public dataset for testing phase retrieval algorithms

41. Acknowledgements
Rice: Richard Baraniuk, Ashok Veeraraghavan, and Manoj Sharma Northwestern: Ollie Cossairt and Sudarshan Nagesh Ohio State: Phil Schniter
Ashok Veeraraghavan
DARPA
ONR
ARO
NSF
Rice University

42. Summary: PR Algorithms Can Image Through Scatterers
TMs for thick and static scattering media
Characterize a scatterer by solving tens thousands of PR problems
“Invert” scattering by solving PR problem
This work: Develop fast phase retrieval algorithm - prVAMP
Release public dataset for testing phase retrieval algorithms

43. Denoise x
Denoise z (z=Ax)
LMMSE estimation of x
LMMSE estimation of z