1. Media Cybernetics www.mediacy.com/aqi/

2. Deconvolution Approaches and Challenges Jonathan Girroir jgirroir@mediacy.com

3. Overview 1.Image Formation 2.Deconvolution 3.Blind and Non-Blind Approaches 4.Spherical Aberration 5.Dealing with Noisy Data 6.Summary and Discussion

4. 1. Image Formation The object The image What happened?

5. 1D - Diffraction

6. 2D - Airy Disk

7. 3D - Point Spread Function (PSF) An impulse response.. Characterizes your microscope.

8. Effects of the PSF An object is a collection of point sources. Mathematical operation called a convolution. The Microscope is a convolution operator. * = ObjectPSFImage

9. 2. Deconvolution Widefield – Single Channel

10. Multiple wavelengths - Widefield

11. Confocal Microscopy Confocal PSF

12. Types of Deconvolution Multiple ways, two categories: “Deblurring” algorithms – No neighbors and nearest neighbors. Subtractive. “Deblurring” algorithms – No neighbors and nearest neighbors. Subtractive. Image restoration algorithms – Inverse filter, non-blind deconvolution, blind deconvolution. Image restoration algorithms – Inverse filter, non-blind deconvolution, blind deconvolution. Trade off between speed and quality

13. 3D Inverse Filter One pass – fast. Limited by noise amplification. Blur removal traded against gain in noise. ImageObjectPSF Imaging : DeblurredImage PSF 1 Inverse Filter:

14. Improving signal to noise

15. 2D Blind deconvolution

16. 2D Blind deconvolution – TIRF

17. When to use deconvolution In widefield fluorescent Microscopy Improve contrast of 2D and 3D images Improve contrast of 2D and 3D images Improve resolution in three dimensions Improve resolution in three dimensions Improve 3D rendering and visualization Improve 3D rendering and visualization

18. When to use deconvolution In confocal microscopy Improve signal to noise ratios Improve signal to noise ratios increase dynamic range increase dynamic range Improve image grayscale resolution (contrast) Improve image grayscale resolution (contrast) Increase axial resolution Increase axial resolution

19. When to use deconvolution Documentation Images and projections will show more detail and greater contrast Images and projections will show more detail and greater contrast Improvements can be striking Improvements can be striking

20. When to use deconvolution Analysis Deconvolution retains quantitative accuracy Deconvolution retains quantitative accuracy Improves contrast facilitating analysis Improves contrast facilitating analysisCountingTracking Area measurements Intensity measurements

21. Why does this work? The microscope is a well understood optical system The behavior of light is well modeled

22. The Point Spread of a Microscope Compare a widefield PSF to a confocal PSF

23. Looking forward to Deconvolution YFP Labeled Motor Neurons (Mouse) Laura Baylor, Harvard University Laboratory of Dr. Jeff Lichtman Emissive Wavelength Emissive Wavelength Numerical Aperture Numerical Aperture Spherical Aberration Spherical Aberration

24. 3. Blind and Non-Blind Approaches Non-Blind Fixed PSF. Fixed PSF. PSF normally measured for each objective, and camera. PSF normally measured for each objective, and camera.Blind Adaptive PSF. Adaptive PSF. PSF normally theoretical. PSF normally theoretical. Requires NA, RI, wavelength, xyz spacing. Requires NA, RI, wavelength, xyz spacing.

25. Non-Blind deconvolution estimates the original object from the measured PSF and the collected data (2D or 3D image). Non-Blind Deconvolution Non Blind Deconvolution

26. Blind deconvolution separates the original object and the point spread from the collected data (2D or 3D image). Blind deconvolution Blind Deconvolution

27. What happens during deconvolution? How it’s done (8/8)

28. Non-Blind Deconvolution Reblurred Initial Object Estimate PSF Original Blurred - UpdateError Improved Object Estimate

29. Blind deconvolution

30. Initial Object Estimate Initial Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error PSF ? ? ? Iteration: 1 - PSF

31. Initial Object Estimate Initial Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error Object Estimate Initial Object Estimate ? ? ? Iteration: 1 - Object

32. Object Estimate Object Estimate Blind Deconvolution PSF Original Blurry Data - Update Error PSF ? ? ? Iteration: 2 - PSF

33. Blind deconvolution

34. Blind vs. Non-Blind Non-blind Must maintain library of PSFs. Must maintain library of PSFs. PSFs need to be optimally measured and noise free. PSFs need to be optimally measured and noise free. Faster because only estimating image. Faster because only estimating image.Blind Does not require imaging of beads. Does not require imaging of beads. Can respond to changes in optical model: Specimen, thermal expansion, aberrations and noise. Can respond to changes in optical model: Specimen, thermal expansion, aberrations and noise.

35. 4. Spherical Aberration Changes in refractive index Incorrect cover slip thickness Thick specimens

36. Results in an asymmetry of the PSF. Adds significantly more blur. Reduces signal. Effects of Spherical Aberration

37. Match refractive indexes. Use a SA correction collar. Bias the PSF with similar aberration and deconvolve. Bias the PSF and allow it to adapt using blind deconvolution. Correction of Spherical Aberration

38. Raw Image Max Intensity 2,614

39. Deconvolution with no SA Max Intensity 15,134

40. Deconvolution with SA Correction Max Intensity 23,938

41. 5. Dealing with Noisy Data Noise comes from: Photon detection noise. Photon detection noise. Gaussian instrument noise. Gaussian instrument noise. Scatter. Scatter. Noise is an issue in: Live cell imaging / low light. Live cell imaging / low light. Confocal imaging. Confocal imaging. Update step is important.

42. Gold’s Method Apply a smoothing every few iterations Quickly can deconvolve images Does not work well for noisy data. Either noise is amplified or small features are removed (over smoothing).

43. Maximum Likelihood Estimation Statistical approach which tests what is the most likely to occur in the specimen using a known noise model. Finds the “most likely” possibility for each portion of the image. Models noise extremely well. Can actually remove noise instead of suppress. MLE is more versatile than Gold’s which just uses a smoothing factor.

44. 6. Summary and Discussion Deconvolution is a powerful tool that can increase resolution, contrast and signal to noise. There are multiple methods available to accommodate a variety imaging needs. Trade offs exist in acquisition and processing.

45. Media Cybernetics www.mediacy.com/aqi/