Image restoration by deconvolution Volker Bäcker Montpellier Rio Imaging http://www.mri.cnrs.fr/ Pierre Travo IFR3 Giacomo Cavalli Frederic Bantignies Patrice Mollard Nicole Lautrédou-Audouy Jean-Michel Poulin volker.baecker@crbm.cnrs.fr

Overview Part 1 Part 2 introduction what is deconvolution ? how does it work ? when should it be used ? Part 2 what are the parameters to know and care about for image restoration by deconvolution?

fluorescence microscopy specimen has to be in focal distance to image 3d specimen move focal plane through specimen creating stack of slides fluorescence microscopy specimen marked with dye that emists light of one wavelength while being stimulated by light of another wavelength Microscope types widefield whole specimen bathed in light confocal image is constructed point by point to keep out out-of-focus light two photon two photons needed to stimulate emission, similar effect as confocal

Example: 2d widefield After deconvolution (same levels) Image from microscope After deconvolution (same levels) Immunostaining on whole mount drosophila Embryo Using an antibody against a nuclear protein

Example 3d confocal Image from microscope After deconvolution

Example: time series 2 photon Image from microscope After deconvolution

The aquired image is not the „real“ image Images are degraded due to the limited aperture of the objective Deconvolution can be used to get an image nearer to the real object by using knowledge of the imaging process and the properties of the microscope Deconvolution can be used for all kinds of fluorescence microscope images: 2D, 3D, time series, widefield, confocal, 2 photon Example of 2D widefield image before and after deconvolution example of 3D image befor and after deconvolution example of confocal image before and after deconvolution

Sources of image degradation Noise Blur Can be handled by image restoration Scatter random distribution of light due to heterogenous refrection index within specimen Glare random distribution of light that occurs within the optical train of the microscope

Causes of image degradation Noise Geben Sie eine Zusammenfassung der momentanen Situation What causes the image degradation

Causes of image degradation Noise Where does the noise come from ? random fluctuations in the signal intensity variation of the incident photon flux interfering signals from electronic system of the captor device

Causes of image degradation Blur Before restoration After restoration

Causes of image degradation Blur Where does the blur come from ? contributions of out-of-focus light to the imaging plane diffraction a result of the interaction of light with matter diffraction is the bending of light as it passes the edge of an object

How does deconvolution work Image restoration Get rid of noise assume random noise with Poisson distribution remove it Get rid of blur Compute real image from sample by applying a model of how the microscope degraded the image deconvolution

Point Spread Function Point spread function (psf) Model of how one point is imaged by microscope Experimental aquired by taking an image of „point like objects“ - beads Alternatevely, point like object present in the acquired image itself can be usedf. Theoretical computed from the microscope and captor parameters

Convolution (Faltung) aquired image = real image convolved with psf Convolution is an integral that expresses amount of overlap of functions as g is shifted over f. N pixel => O(N*N) operations to compute it i(x) : aquired image f(x) : object function g(x) : point spread function

Fourier Transform (FT) Signal can be represented as sum of sinoids FT transforms from spacial to frequency domain

Fourier transform (FT) Convolution theorem <=> i(x) : aquired image f(x) : object function g(x) : point spread function I fourier transform of i F fourier transform of f G fourier transform of g * Object function psf Fourier transform (FT) FT inverse FT FT can be computed in O(n * log n) Object function convolved with psf

Deconvolution <=> Deconvolution: find object function f for given image i and psf g Unfortunatly it is not practicable to compute G has zeros outside certain regions Setting F zero for these would create artefacts In practice there is noise N/G would amplify noise It's not possible to reconstruct the real object function

Deconvolution algorithms Solution Find an algorithm that computes a function f' so that f' estimates f as good as possible works in the presence of noise Different deconvolution algorithms exist In general best for fluorescent microscopy: (Classical) Maximum Liklihood Estimation - MLE

Maximum Likelihood Estimation Tries to optimise f' iteratively The basic principal is (but there's more to it) g(i|j) : psf - the fraction of light from true location j that is observed in pixel i Fraction of light from pixel j that hit other pixels Fraction of light from other pixels that hit pixel j Richardson and Lucy R-L Iteration

fraction of light from others 0,3 1 0,1 0,2 A B C D 6 5 3 4 Denominator: get rid of foreign light that hit me 1 C3 + 0.3 C4 + 0.2 B3 Numerator realign my light to me 1 C3 + 0.1 C4 + 0.2 B3 1 2 3 psf 4 image 5 * [5*1 / (5*1 + 0.3*4 + 0.2*6) + 0,1*4 /(5*1 + 0.3*4 + 0.2*6) + 0,2*6/(5*1 + 0.3*4 + 0.2*6)] 5 * [5 / 7.4 + 0.4/7.4 + 1.2/7.4] fraction of light lost 5 * [(5 + 0.4 + 1.2)/7.4] 5 * [6.6 /7.4] 5 * 0.891891 New estimate aquired image last estimate 4.459459 last estimate fraction of light from others

Summary and conclusions 1 image from microscope is degraded it contains noise and blur blur can be described as a convolution of object function and psf image nearer to the object function can be obtained by image restoration yielding higher resolution and better contrast MLE is a deconvolution algorithm approriate for fluorescent microscope images imaging process is not finished finished without deconvolution do it whenever high quality images are needed

Image restoration in practice Many deconvolution software packages are commercially available They use various types of deconvolution algorithms In addition to these algorithms, they might incorporate other imaging tools, such as filters of different kinds. Moreover, different types of algorithms may introduce or not some « assumptions » concerning the image sent to restoration. In general, it is important to test the software. One basic « rule of thumb » is also that the restoration should respect the acquired image in terms of objects visible and of their relative intensity. Objects « appearing », « disappearing » or changing relative intensity with respect to neighboring structures are diagnostic of problems. These problems might be due to the setting of relevant parameters or, in the worst case, of poor quality of the software

Image restoration using the huygens2 software from SVI http://www.svi.nl/ - website of Scientific Volume Imaging (SVI) It is the software used at the Institute of Human Genetics

Relevant parameters in deconvolution Setting Microscope parameters microscope type widefield and multipoint confocal work with ccd camera single point confocal and two photon work with photomultiplier different point spread functions if you don't know Ask your imaging facility and look at the specifications of your microscope

Microscope parameters Numerical aperture measure of ability to gather light and resolve fine specimen detail at a fixed object distance higher magnification doesn't yield higher resolution, higher NA does Maximal value written on objective Can't be larger than the the refractive index n of the medium

Sampling theorem Imaging converts an anlog signal into a digital signal When converting an analog signal into a digital signal the sampling theorem applies Nyquist-Shannon sampling theorem “the sampling interval must not be greater than one-half the size of the smallest resolvable feature of the optical image” sampling at nyquist rate means using exactly this interval sampling interval is the pixel size in our image

Undersampling and oversampling loss of information aliasing artefacts over sampling higher computation times and storage requirements longer acquisition times, photobleaching. under sampling example. An object of a given shape (dashed line) can be interpreted as a different shape (thick line) if too few points are acquired along any of the x,y,z axes

Changing the Numerical Aperture (NA) for widefield / two photon huygens2 allows under/oversampling within a range at the borders of this range deconvolution can be done but results are not good In this case better results when “lying” about NA if sampling size not in range change NA nyquist sample size

Microscope parameters Excitation and emission wavelength fluorescent dye absorbs light of one wavelength and emits light of another wavelength filter cubes are used to ensure that only light of a wanted wavelength passes. exitation and emission wavelengths depend on the cube used GFP 473, 525

Microscope parameters The objective magnification used determines the pixel size in the image ccd camera Pixel size = ccd element size / magnification (eventually modified by other parameters) photomultiplier pixel size depends on resolution and magnification

Microscope parameters Refractive index n of the objective medium oil 1,51500 water 1,33810 air 1,00000 Should match the refractive index of the sample medium Otherwise Magnification error in axial direction Spherical aberration (psf deteriorates with increasing depth)

Microscope parameters Cmount factor adaptor that attaches the camera to the microscope might contain additional optic that changes the overall magnification and therefore the pixel size value is 1 if no additional optic present

Microscope parameters Tube factor the tube might contain additional optics to change the tube length this changes the overall magnification and therefore the pixel size

Microscope parameters sample medium refractive index n default (all media for example water) 1,33810 liquid Vectashield (not polymerized) 1,49000 90-10 (v:v) glycerol - PBS ph 7.4 1,49000 prolong antifade 1,4 limits the NA and therefore the possible resolution

Captor parameters size of the unitary ccd captor image sensor of the camera ccd – charge coupled device diodes that convert light into electrical charge property of the camera Coolsnap 6450 nm Micromax 6700 nm For photomultiplier the pixel size is asked see table in help pages

Captor parameters Binning take nxn elements as one more light per pixel reduces noise higher signal to noise ratio lower resolution

Captor parameters in case of XZY in case of time series z step size time interval

Captor parameters in case of confocal pinhole radius pinhole keep out out of focus light pinhole either fixed or adjustable Backprojected radius in nm Size of pinhole as it appears in the specimen plane size should match airy disk (2d psf) size 6.66 for LSM510

task parameter Style of processing step process image slide by slide converts stack into time series for processing converts result back into stack volume use 3d information step combined do step processing followed by volume processing with fixed parameters

Full restoration parameters signal/noise ratio the ratio of signal intensity to noise intensity high noise case can be measured in the image Single photon hit intensity find low intensity voxels from one photon hit – add values – subtract background Max voxel value value of brightest voxel low noise case single photon hits can´t be seen rough guess is sufficient

Full restoration parameters background offset empty regions should be black but contain some light in reality subtract mean background to see object clearly

Full restoration parameters number of iterations too low optimal restoration not yet achieved too high takes longer to compute some signal may be removed Usually between 30-70

Summary and conclusions 2 deconvolution should be used to obtain high quality images for all kind of fluorescent microscope images parameters of the imaging system have to be entered to create a model of the image degradation

End of presentation volker.baecker@crbm.cnrs.fr Giacomo.Cavalli@igh.cnrs.fr

Links participants literature Montpellier RIO Imaging http://www.mri.cnrs.fr/ IFR3 / CCIPE http://www.montp.inserm.fr/ifr3.htm IGH http://www.igh.cnrs.fr/ CRIC http://www.iurc.montp.inserm.fr/cric/index.htm literature Introduction to Fluorescence Microscopy http://www.microscopyu.com/articles/fluorescence/fluorescenceintro.html How does a confocal microscope work? http://www.physics.emory.edu/~weeks/confocal/ Two-Photon Fluorescence Microscopy http://www.fz-juelich.de/ibi/ibi-1/Two-Photon_Microscopy/ Deconvolution in Optical Microscopy http://micro.magnet.fsu.edu/primer/digitalimaging/deconvolution/deconintro.html

Links literature Diffraction of Light http://micro.magnet.fsu.edu/primer/java/diffraction/basicdiffraction/ Image restoration: getting it right http://www.svi.nl/support/talks/GettingItRight.pdf Image Restoration in Fluorescence Microscopy http://www.ph.tn.tudelft.nl/Publications/PHDTheses/GMPvanKempen/thesis_kempen.html Image restoration in one- and two-photon microscopy http://www.svi.nl/support/talks/Vancouver97.pdf Introduction to Convolution http://cnx.rice.edu/content/m11542/latest/ An Introduction to Fourier Theory http://aurora.phys.utk.edu/~forrest/papers/fourier/ A Self Contained Introduction to Fourier Transforms http://www.doc.eng.cmu.ac.th/course/cpe496/notes/fourier.pdf Convolution theorem http://www.fact-index.com/c/co/convolution_theorem.html

Links literature Three-Dimensional Imaging by Deconvolution Microscopy Article ID meth.1999.0873, available online at http://www.idealibrary.com on IDEAL Deconvolution of confocal images of dihydropyridine and ryanodine receptors in developing cardiomyocytes http://www.sfu.ca/~tibbits/research/JAP04.pdf Maximum likelihood estimation via the ECM algorithm: A general framework http://www.jbs.agrsci.dk/~lfo/talks/ECM_talk.pdf The influence of the background estimation on the superresolution properties of non-linear image restoration algorithms http://www.ph.tn.tudelft.nl/People/lucas/publications/1999/SPIE99GKLV/SPIE99GKLV.pdf Numerical Aperture and Resolution http://micro.magnet.fsu.edu/primer/anatomy/numaperture.html User guide for Huygens Professional and Deconvolution Recipes http://www.svi.nl/download/ Digital Image Sampling Frequency http://www.olympusmicro.com/primer/java/digitalimaging/processing/samplefrequency/index.html Filter Cubes http://www.olympusmicro.com/primer/techniques/fluorescence/filters.html Filters for fluorescence microscopy http://www.nikon-instruments.jp/eng/products/option/index1.aspx

Links literature Immersion Media http://www.olympusmicro.com/primer/anatomy/immersion.html How Digital Cameras Work http://electronics.howstuffworks.com/digital-camera2.htm Pixel Binning http://micro.magnet.fsu.edu/primer/digitalimaging/concepts/binning.html CCD Signal-To-Noise Ratio http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/