Media Cybernetics

Deconvolution Approaches and Challenges Jonathan Girroir

Overview 1.Image Formation 2.Deconvolution 3.Blind and Non-Blind Approaches 4.Spherical Aberration 5.Dealing with Noisy Data 6.Summary and Discussion

1. Image Formation The object The image What happened?

1D - Diffraction

2D - Airy Disk

3D - Point Spread Function (PSF) An impulse response.. Characterizes your microscope.

Effects of the PSF An object is a collection of point sources. Mathematical operation called a convolution. The Microscope is a convolution operator. * = ObjectPSFImage

2. Deconvolution Widefield – Single Channel

Multiple wavelengths - Widefield

Confocal Microscopy Confocal PSF

Types of Deconvolution Multiple ways, two categories: “Deblurring” algorithms – No neighbors and nearest neighbors. Subtractive. “Deblurring” algorithms – No neighbors and nearest neighbors. Subtractive. Image restoration algorithms – Inverse filter, non-blind deconvolution, blind deconvolution. Image restoration algorithms – Inverse filter, non-blind deconvolution, blind deconvolution. Trade off between speed and quality

3D Inverse Filter One pass – fast. Limited by noise amplification. Blur removal traded against gain in noise. ImageObjectPSF Imaging : DeblurredImage PSF 1 Inverse Filter:

Improving signal to noise

2D Blind deconvolution

2D Blind deconvolution – TIRF

When to use deconvolution In widefield fluorescent Microscopy Improve contrast of 2D and 3D images Improve contrast of 2D and 3D images Improve resolution in three dimensions Improve resolution in three dimensions Improve 3D rendering and visualization Improve 3D rendering and visualization

When to use deconvolution In confocal microscopy Improve signal to noise ratios Improve signal to noise ratios increase dynamic range increase dynamic range Improve image grayscale resolution (contrast) Improve image grayscale resolution (contrast) Increase axial resolution Increase axial resolution

When to use deconvolution Documentation Images and projections will show more detail and greater contrast Images and projections will show more detail and greater contrast Improvements can be striking Improvements can be striking

When to use deconvolution Analysis Deconvolution retains quantitative accuracy Deconvolution retains quantitative accuracy Improves contrast facilitating analysis Improves contrast facilitating analysisCountingTracking Area measurements Intensity measurements

Why does this work? The microscope is a well understood optical system The behavior of light is well modeled

The Point Spread of a Microscope Compare a widefield PSF to a confocal PSF

Looking forward to Deconvolution YFP Labeled Motor Neurons (Mouse) Laura Baylor, Harvard University Laboratory of Dr. Jeff Lichtman Emissive Wavelength Emissive Wavelength Numerical Aperture Numerical Aperture Spherical Aberration Spherical Aberration

3. Blind and Non-Blind Approaches Non-Blind Fixed PSF. Fixed PSF. PSF normally measured for each objective, and camera. PSF normally measured for each objective, and camera.Blind Adaptive PSF. Adaptive PSF. PSF normally theoretical. PSF normally theoretical. Requires NA, RI, wavelength, xyz spacing. Requires NA, RI, wavelength, xyz spacing.

Non-Blind deconvolution estimates the original object from the measured PSF and the collected data (2D or 3D image). Non-Blind Deconvolution Non Blind Deconvolution

Blind deconvolution separates the original object and the point spread from the collected data (2D or 3D image). Blind deconvolution Blind Deconvolution

What happens during deconvolution? How it’s done (8/8)

Non-Blind Deconvolution Reblurred Initial Object Estimate PSF Original Blurred - UpdateError Improved Object Estimate

Blind deconvolution

Initial Object Estimate Initial Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error PSF ? ? ? Iteration: 1 - PSF

Initial Object Estimate Initial Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error Object Estimate Initial Object Estimate ? ? ? Iteration: 1 - Object

Object Estimate Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error PSF ? ? ? Iteration: 2 - PSF

Blind deconvolution

Blind vs. Non-Blind Non-blind Must maintain library of PSFs. Must maintain library of PSFs. PSFs need to be optimally measured and noise free. PSFs need to be optimally measured and noise free. Faster because only estimating image. Faster because only estimating image.Blind Does not require imaging of beads. Does not require imaging of beads. Can respond to changes in optical model: Specimen, thermal expansion, aberrations and noise. Can respond to changes in optical model: Specimen, thermal expansion, aberrations and noise.

4. Spherical Aberration Changes in refractive index Incorrect cover slip thickness Thick specimens

Results in an asymmetry of the PSF. Adds significantly more blur. Reduces signal. Effects of Spherical Aberration

Match refractive indexes. Use a SA correction collar. Bias the PSF with similar aberration and deconvolve. Bias the PSF and allow it to adapt using blind deconvolution. Correction of Spherical Aberration

Raw Image Max Intensity 2,614

Deconvolution with no SA Max Intensity 15,134

Deconvolution with SA Correction Max Intensity 23,938

5. Dealing with Noisy Data Noise comes from: Photon detection noise. Photon detection noise. Gaussian instrument noise. Gaussian instrument noise. Scatter. Scatter. Noise is an issue in: Live cell imaging / low light. Live cell imaging / low light. Confocal imaging. Confocal imaging. Update step is important.

Gold’s Method Apply a smoothing every few iterations Quickly can deconvolve images Does not work well for noisy data. Either noise is amplified or small features are removed (over smoothing).

Maximum Likelihood Estimation Statistical approach which tests what is the most likely to occur in the specimen using a known noise model. Finds the “most likely” possibility for each portion of the image. Models noise extremely well. Can actually remove noise instead of suppress. MLE is more versatile than Gold’s which just uses a smoothing factor.

6. Summary and Discussion Deconvolution is a powerful tool that can increase resolution, contrast and signal to noise. There are multiple methods available to accommodate a variety imaging needs. Trade offs exist in acquisition and processing.

Media Cybernetics