9.4 Enhancing the SNR of Digitized Signals stepping and averaging compared to ensemble averaging creating and using Fourier transform digital filters removal of Johnson noise and signal distortion using a running average, lead-lag filter, and a brick-wall filter removal of interference noise using a rectangular filter and a missing-one-frequency filter 9.4 : 1/15

Stepping and Averaging Consider an experiment that attempts to improve the SNR of a spectrum by averaging. The simplest strategy is called stepping and averaging. Stepping and Averaging start at 400 nm, digitize the detector output at some rate (10 Hz), and average long enough (10 s) to reach a target SNR step to the next wavelength (401 nm) and repeat the data collection the overall time is determined by the time spent averaging at each wavelength and the number of steps, e.g. for 10 s/step  101 steps = 1,010 s with Johnson noise this procedure is perfectly fine the long collection times make the experiment susceptible to 1/f noise the 1/f noise will distort the spectrum since it varies from one wavelength to the next, depending upon the total time required to collect data 9.4 : 2/15

Susceptibility to 1/f Noise The 1/f noise at the right was added to the data while stepping and averaging (10,100 data points were collected). Stepping and averaging gave the spectrum, where the wavelengths near 400 nm had the 1/f noise toward the left of the above graph, and wavelengths near 500 nm had the 1/f noise toward the right. Note that the low frequency drift has not been removed. 9.4 : 3/15

Ensemble Averaging Ensemble averaging start at a wavelength and collect one data value step to the next wavelength and collect one data value repeat the process until one spectrum has been collected - this is called an ensemble of data reset the wavelength to the original starting value and collect a second spectrum in the same way collect many ensembles, store them and average at a later time this reduces the effect of 1/f noise since the time of concern is only that required to collect one ensemble (10.1 s) and not the entire data set (1,010 s) The spectrum was obtained by averaging 100 ensembles. Note the reduction in 1/f noise in comparison to the spectrum obtained by stepping and averaging. 9.4 : 4/15

Fourier Transform Filters used to process a vector of data that has already been digitized no matter how the data were obtained, they are treated as if they are amplitude versus time Fourier transform filters are based on signal-to-noise enhancement by multiplication with a spectral transfer function, Ffilter(f ) filtering is implemented by convolution with a temporal transfer function, Ffilter(t), where Ffilter(t)  Ffilter(f ) the normalizer, norm, is required when the digital filter, Ffilter(t), does not have a unit area the value of norm is most often, but not always, given by the integral of Ffilter(t) 9.4 : 5/15

Discrete Convolution when using data vectors and not functions, the integral is replaced by a summation the variables t and t' are replaced by subscripts of the vectors let the data vector have subscripts, i = 0 ... D let the filter vector have subscripts, j = 0 ...F, where F < D and where F+1 is an odd number the smooth vector will have subscripts, ii = F/2 ... (D-F/2), where the first and last F/2 values will be indeterminate (cannot be computed) working out the subscript expression is easiest if the filter vector is reversed before performing the convolution the value of norm is most often, but not always, given by the sum of the elements of the vector, filter when graphing the smoothed vector, don't plot the first and last F/2 points 9.4 : 6/15

Symbolic Convolution Example Look at a symbolic example where F = 4 and D = 20. Thus j = 0 ... 4, i = 0 ... 20, and ii = 2 ... 18. f0 f1 f2 f3 f4 d0 d1 d2 d3 d4 d5 d6  smooth2 = (f0d2-2+0 + f1d2-2+1 + f2d2-2+2 + f3d2-2+3 + f4d2-2+4)/ norm smooth3 = (f0d3-2+0 + f1d3-2+1 + f2d3-2+2 + f3d3-2+3 + f4d3-2+4)/ norm smooth4 = (f0d4-2+0 + f1d4-2+1 + f2d4-2+2 + f3d4-2+3 + f4d4-2+4)/ norm 9.4 : 7/15

Spectrum with Johnson Noise Consider the spectrum below, which has superimposed Johnson noise with s = 0.05 V. The noise free spectrum has a peak at 250 nm with an amplitude of 1 V. Knowing that the noise is Johnson permits the use of almost any digital filter. The primary consideration will be the trade of SNR against distortion. Success will be measured by the increase in the SNR and the minimization of distortion. 9.4 : 8/15

Rectangular Filter (Running Average) SNR improvement is due to multiplication by a sinc function in the frequency domain, which reduces the integrated Johnson noise. The filter width can be varied to trade SNR against distortion. The expected SNR improvements are 3.3 and 10. The distortion is symmetric because the filter is an even function. The distortion lowers the peaks and fills the valleys. 9.4 : 9/15

RC Lead-Lag Filter The lead-lag filter is an even exponential, which multiplies the spectrum by a Lorenztian. Since the Lorentzian drops more slowly than a sinc function, it is less distorting. Both filters had 101 elements, with the amount of distortion controlled by the time constant. Note that the distortion is symmetric, unlike a normal RC low pass filter. Lead-lag filters are difficult to implement in hardware unless the signal is composed of high frequencies. 9.4 : 10/15

Brick-Wall Filter A brick-wall filter is so named because it will truncate the spectrum at some specified frequency. To have this property the temporal data must be convolved with a sinc function. Two brick-wall filter outputs are shown below. Both are 101 elements long. The t 0 = 10 sinc function spacing has almost no distortion but not such a great SNR. The t 0 = 30 filter has some distortion because the sinc function is being truncated at large amplitudes. Brick wall filters are usually large, thus require large data sets. 9.4 : 11/15

Spectrum with Interference Noise Consider the spectrum below, which has superimposed cosine interference noise with a period of 101 units and an amplitude of 0.1. Knowing the noise is interference permits the use of Fourier filters specifically designed to remove it. Success will be measured by reduction in the interference amplitude and the minimization of distortion. 9.4 : 12/15

101-Element Rectangle The Fourier transform of a 101-element rectangle is a spectral sinc function having a node at the interference frequency. This filter will also remove all harmonics of the interference which could be good (60, 120, 180 Hz interference) or bad (important signal frequencies). The interference has been removed but the sinc function is distorting the signal frequencies as seen with Johnson noise. 9.4 : 13/15

Missing-One-Frequency Filter An alternative filter is the one described with Fourier transform pictures, which consisted of -2cos(2pt/101) plus an impulse at t = 0. This single-frequency filter only works well when it extends for many cycles of the cosine. The spectrum below was obtained by filtering with a 401 element filter so that several cycles could be obtained. The result has very little distortion, however, many data points needed to be dropped on either end of the spectrum. The filter will not remove Johnson noise. 9.4 : 14/15

Digital Filter Transfer Function Examine the signal and noise spectra separately. Develop a multiplicative transfer function which enhances the signal-to-noise ratio. Fourier transform the spectral transfer function to obtain the corresponding temporal transfer function. Convolve the measured data with the temporal transfer function. This strategy works quite well as long as a signal spectrum can be obtained that has little noise, and a noise spectrum can be obtained that has little signal. Performance is usually enhanced by increasing the length, F, of the temporal transfer function (digital filter). However, an increase in filter length means that more elements of the data vector cannot be processed because convolution drops the first and last F/2 points. 9.4 : 15/15