Basics of Deconvolution Presented by: Andrew Molnar Senior Software Engineer, Media Cybernetics

What is Deconvolution? Deconvolution is an image restoration operation that assists in reversing the blurring effects of an optical system. It is distinct from deblurring (nearest neighbors, unsharp masking, etc.) in that it reassigns, rather than removes, out of focus light.

Why Deconvolution? The image is not the sample. The collected image is distorted by the optics, illumination, noise, as well as the sample itself. All images have distortions and limitations, no matter how good they look. Deconvolution reveals structure obscured by the distortions inherent in any imaging process.

Object convolved with the PSF, added to Noise, equals the Image Image Formation Object PSF Noise Image  + = Object convolved with the PSF, added to Noise, equals the Image Each object in the optical path causes some blur. The blur of a single point of light is called the point-spread function (PSF). The PSF, applied to all points in the image, forms the observed image. Additionally, noise is introduced through a combination of ambient factors and imaging hardware.

Light Diffraction – Airy Disc Light through an objective converges and interferes at the image plane Interference pattern is called an Airy pattern At the center is an Airy disc

Resolution and Blur Structures separated by more than an Airy disc are resolvable. Any structures separated by less than an Airy disc are blurred together and appear as one structure

Deconvolution Overview Acquire microscope data Generate/Collect PSF Deconvolve

Acquisition Criteria Sample interval (Nyquist): >2x deconvolved resolution XY: Z: 100x 1.4NA oil, @ 550nm: XY < 0.07 mm, Z < 0.3 mm Acquire slices several DOF above/below sample Minimize refractive index mismatch Best case: immersion RI = sample RI Do not saturate pixels! The best results come from the best data     Recommend sampling 2.3 to 2.5x the highest resolution Z slices < 0.25 are not worthwhile even with the best objectives

Deconvolution Overview Acquire microscope data Generate/Acquire PSF Deconvolve

Determining the PSF Just as important (perhaps more so) as choosing the correct deconvolution approach Measured PSF: Acquire a point-object image to use as a PSF Theoretical PSF: Generate a model PSF from optical parameters

Measured PSF: How to collect Image a bead with a size ideally smaller than the resolution limit of the microscope Acquisition must be done under the same conditions as the sample Pixel calibration mismatches can be corrected Can be imaged simultaneously with the sample: include a bead on the slide, isolated from the sample

Theoretical PSF: What affects the shape? Scope Modality Must be a model supported by the software to generate a theoretical PSF Example: AutoQuant X3 supports: Widefield Fluorescence Transmitted-Light Brightfield Laser Scanning Confocal Spinning Disk Confocal Multi-Photon Fluorescence Unknown modalities can still be deconvolved using a measured PSF!

Theoretical PSF: What affects the shape? Lens parameters Numerical Aperture Refractive Index of Lens Immersion Pixel Calibration Emission Wavelength Specimen Parameters Refraction Index of Specimen Embedding Specimen Distance from Coverslip Spherical Aberration XY XY XZ XZ Widefield Confocal

Spherical Aberration Caused by Solve by Match RI Changes in refractive index Thick specimens Incorrect cover slip thickness Worse deeper in the sample Solve by Match RI SA correction collar Use biased PSF and deconvolution

SA: PSF changes with depth SA gets worse deeper into sample Most deconvolution algorithms assume constant PSF Calculate/Collect PSF at same depth (z) as objects in the sample.

Choosing a PSF Theoretical: Best for supported modalities when optical parameters are known, or when a good bead image is not available. Measured: Best when a good bead image is available, or for unsupported modalities (STED, SI, etc.).

Deconvolution Overview Acquire microscope data Generate/Acquire PSF Deconvolve

Approaches to Deconvolution Inverse Filter Constrained Iterative Statistical Constrained Iterative Non-blind Blind

Inverse Filter: Solve for Object directly 1 ? =  Object Object becomes image convolved with inverse of PSF Weiner, Regularized Least Squares. Pros Relatively quick (non-iterative) Reasonable results when signal to noise is high Cons Does not address noise directly Breaks down with inaccurate PSF

Constrained Iterative Deconvolution Algorithms Optimization approach Work by improving solution through repeated iterations. Constraints Non-negativity Filter Gaussian Noise Jansen-Van Cittert, Least Squares.

Statistical Constrained Iterative Deconvolution Optimization is done using statistical criteria (Maximum Likelihood Estimation) Better noise handling Poisson noise Most likely estimate of object given that photon emission is Poisson. Most (if not all) modern deconvolution packages use statistical constrained iterative methods

Statistical Constrained Iterative: Blind and Non-Blind Adaptive PSF (Blind) Both image and PSF estimate are changed as part of iterative optimization Can adapt to imperfections in the microscope Can adapt to specimen refractive index Handles spherical aberration well Fixed PSF (Non-blind) PSF is fixed; remains unchanged with each deconvolution iteration Only the image estimate is updated at each iteration More consistent processing across volumes

Iterative Deconvolution (Fixed PSF) Object Estimate PSF Image Estimate Image - Update Constraints Error First Iteration

Deconvolution Data Improvements Increased resolution Higher contrast, reduced haze Noise suppression Resolution increase of ~30% in XY, roughly equivalent to confocal depending on sample structure Richard Cole, NYS Department of Health, Biggs Laboratory- Wadsworth Center, Albany NY 12201

Improvements are Additive Deconvolve confocal data Improve resolution Improve S/N: Noise without a PSF will be suppressed

Uses - Increase Contrast and Resolution

Uses – Measurement/Classification AA erythrocyte Fuyuki Tokumasu National Institute of Allergy and Infectious Diseases Journal of Cell Science 118 ,1091-1098

Uses - Analysis Raw Image Deconvolved Threshold Dan Mulvihill Cell Developmental Biology Group University of Kent

Some Side Effects of Deconvolution Increased dynamic range 12-bit unprocessed image may require 16 or 32 bits to store the result Caused by reassignment of light to point sources Image may appear darker overall Bright spots are much brighter, but are much fewer in number

Potential Issues Generally blurred or noisy results Check noise estimate Remaining blur, “ringing” (see individual slices) Try additional or fewer iterations Examine XZ, YZ slices If an obvious PSF blur remains, the PSF may be inaccurate

Frequently Asked Questions How do I know my data isn’t “Photoshopped”? Deconvolving known structures (beads, microtubules, etc) demonstrates that deconvolution accurately restores those structures; within the information limits of the microscope Is deconvolved data quantitative? Note: fluorescence imaging does not provide absolute intensities outside single molecule or internal reference (ratio, FRET) imaging, it gives relative intensities Deconvolution is more quantitative than the raw data: lower noise, less out of focus haze contamination, more accurate spatial information (edges, positions)

Validation Publication Calibration of Wide-Field Deconvolution Microscopy for Quantitative Fluorescence Imaging J Lee, T Wee, C Brown – Journal of Biomolecular Techniques – 2014 - ABRF Raw Deconvolved Follows “best practices” for image collection. Measures multiple sets of bead collections before and after deconvolution. Intensity analysis determines that “… the deconvolved images clearly indicate that deconvolution using the AutoQuant X3 blind deconvolution algorithm preserves the relative quantitative intensity data.” Volume analysis finds that “deconvolution decreased the measured volume to 70% of the original image volumes, which is much closer to the expected volume… based on the actual microsphere size.”

Questions?