Massey University Image Resolution Improvement from Multiple Images Donald Bailey Institute of Information Sciences and Technology Massey University Palmerston North NEW ZEALAND

Overview Describe the resolution improvement process Describe the results of my investigations Investigations in 1 dimensions –Super-resolution of bar codes Registration in 2 dimensions –Comparison of registration methods –Detailed description of predictive interpolation

Description of the Problem Given a set of related independent low resolution images, combine these together to construct a single high resolution image –output has more detail than any of the input images

Resolution Improvement Resolution limited by number of pixels –Resolution depends on sampling density An ensemble of images: –Each image provides separate samples –Potentially higher sampling density Reconstruction steps: –Register images –Resample ensemble –Inverse filter

Sampling Requirements Images must be sub-sampled If sample rate is greater than Nyquist rate –Can reconstruct the image at any desired resolution –A single image contains all information –Can only improve the signal to noise ratio If sample rate is less than Nyquist rate –Each individual image is aliased –Cannot obtain higher real resolution from single image –Resampling the ensemble untangles the aliased information

One Dimension Example “Super-Resolution of Bar Codes”, D.G. Bailey, Journal of Electronic Imaging, 10 (1), pp ( January 2001). Problem: How can we read this bar code?

Information content of UPC Bar Codes 12 digits, each 7 units wide Guard bands Total width is 95 units Each bar or space is 1-4 units wide Broadband frequency spectrum Centre of main lobe contains required data 6 digits guard bands

Super-resolution procedure A tilted 2D bar code image provides an ensemble of independently sampled 1D images Register low-resolution images –gives relationship between individual images Resample the ensemble at a higher rate –creates a high resolution image Remove system effects –reduces the sampling blur and effect of camera electronics

Registration Determines the offset between rows Phase shift in frequency domain proportional to linear offset and spatial frequency Procedure: –Fourier Transform each row, keep phase –Subtract phase of first row from each row –Unwrap phase image –Discard higher frequencies –Least squares fit to calculate offset per row Input image Phase image

Resampling By interleaving samples from different rows increase the sample rate. Original image samples New image samples Selected sample rows

Resampling By interleaving samples from different rows increase the sample rate. Original image samples New image samples Selected sample rows

Resampling Increase sample rate by an integer multiple of original sample rate Select rows with offsets nearest the desired sample positions 4 x sample rate

Practical limitations From synthetic image From actual image 0 Spatial frequency Amplitude  0 /2  0  0  0 /2 0 Spatial frequency Amplitude  0 /2  0  0  0 /2

Practical limitations Real bar code limitations –Ink smearing means bar and space widths not exact –Smears the envelope in the frequency domain Image capture degradations Object Camera angle Lens system Sensor Digital Image Video signal Video frame grabber Camera electronics

Practical limitations Image distortions –Perspective distortion from camera angle –Lens distortion Lens point spread function –Spatially variant low pass filter Image sensor –area sampling - low pass filter –aliasing Camera electronics –low pass filter, perhaps with high frequency emphasis Frame grabber –sampling (more aliasing)

Removing system effects Aliasing is not a problem –it is actually necessary for higher resolution reconstruction –resampling the ensemble untangles the aliased information Main effects are the low pass filter characteristics –lens point spread function –area sampling in sensor –smoothing filter camera electronics –anti-alias filter in frame grabber

Removing system effects 0 Spatial frequency Amplitude  0 /2  0  0  0 /2 Assume no distortion, and no spatial variation in the low pass filter characteristic Estimate the system response by comparing synthetic and actual reconstructed images Remove using an inverse linear filter

Results Resampled ensemble System response removed Straightened and averaged Thresholded Original image

Bar Code Conclusions A two-dimensional image of a bar code tilted slightly provides an ensemble of related one-dimensional images The low resolution images must be aliased It is necessary to compensate for limitations in the image capture system Analogue video cameras make more complex –Image sampled twice –Additional analogue filters Modest gains in resolution are achievable

Extending to 2 Dimensions Problem is considerably more complex Require multiple 2 dimensional images –Captured at different times –Motion is a limitation Registration more complex –Bar code images all had constant offset per row –In 2D every image is independent, with 2D offset Resampling more complex –Need more images for same improvement 4 images to improve resolution by 2

Registration in 2 Dimensions Requirements –Accurate sub-pixel offset between images –Work directly on low resolution images –Insensitive to aliasing –Tolerates a low level of noise –Does not rely on particular objects in the image –For practical use, must be fast Conventional approach to sub-pixel registration –Interpolate images to chosen high resolution Increased data volume slows this method down –Perform pixel accuracy registration Requires a search

Sub-Pixel Registration Methods Phase based methods –Similar to 1D case, but extended to 2D Determine fit surface on integer grid –Interpolate this to find optimum fit –Correlation methods –Difference methods Predictive interpolation –A new method that turns problem around Other approaches –Rely on locating objects or edges within the image

Phase Methods An offset in the image domain corresponds to a phase shift in the frequency domain Procedure –Window the image and reference –FFT and keep the phases –Unwrap the phase difference –Weighted least squares fit of a plane to the phase

Correlation - Pixel Accuracy Multiply an offset image by a reference Accumulate product in the overlap region Normalise by the average pixel value –Prevents bias if the image has a gradient Frequency domain correlation

Correlation - Sub-pixel Accuracy i Correlation peak c(i) i pk i 0 i 1 i  1 c  1 c 1 c 0 Perform pixel level correlation first Interpolate to find peak to sub-pixel accuracy Expect correlation peak to be a pyramid –Only strictly true with regions of uniform value –Approximately true if there are step edges Width of pyramid is twice smallest feature width –Only local information should be used

Can be shown to be related to correlation Subtract an offset image from a reference Accumulate difference in the overlap region Minimum gives offset to nearest pixel For sub-pixel accuracy –Expect minimum to be an inverted pyramid (locally) –Interpolate to find minimum using previous method Difference Methods

Other Registration Methods Centre of gravity –Segment objects from background –Centre of gravity of objects to sub-pixel accuracy Line fitting –Detect lines or edges –Fit a line or curve to detected points Requires knowledge of contents of image Accuracy limited by size of object / edge and accuracy of segmentation / detection

Predictive Interpolation Turns the problem around –Predicts the pixel values as a function of those in a reference image –Uses the bilinear interpolation equation as a linear predictor –Subject to the constraint: –Then offset is:

Predictive Interpolation Procedure –Requires the image to be pre-registered to nearest pixel Can use a search, or hierarchical registration to do this –Determine coefficients A xx that minimise the error Uses weighted least squares Weight each point with standard deviation of its 4 references –From coefficients, get offset directly Properties –Very fast - single pass if registered to nearest pixel –Very good accuracy - typically better than 5% of a pixel

Evaluating the Registration Requires a set of images with known offsets –Start with a single high resolution image –Simulate capturing with a lower resolution camera by filtering and subsambling –Add random Gaussian noise to each low resolution image

Evaluation Procedure Measure offset between each pair of images –36 offset measurements for 9 images Enforce consistency between measurements –Only 8 independent offsets –Will reduce errors up to Error is difference between expected offset and measured offset –Average to give the RMS registration error

Results Structured image No noise: –All worked well Noise sensitivity: –Phase method only few frequencies –Predictive method weighting gives only few measurement A

Results Low detail image No noise –Phase is best –Others adequate Noise sensitivity: –Predictive method sensitive to noise B

Results Medium detail image No noise –Phase and Predictive significantly better Noise sensitivity: –Correlation noise insensitive –Difference noise insensitive C

Results Low detail text No noise –Phase and predictive significantly better Noise sensitivity: –Predictive method improves with low noise D

Results High detail text –Significant aliasing No noise –Correlation and Difference are significantly poorer –Phase and Predictive give excellent results E

Summary of Results Correlation and Difference methods –Similar results –Performance deteriorates with increasing detail –Relatively insensitive to noise Phase and Predictive methods –Best overall with % pixel accuracy Includes enforcing consistency Predictive method –Results almost independent of image –Improved with addition of small amounts of noise

Reconstruction Low resolution Resampled Inverse filtered

Conclusions of Comparison Correlation and difference methods –Insensitive to noise –Perform poorly in presence of high detail –Pyramidal interpolation method breaks down Phase and predictive methods –More sensitive to noise –Suitable for registration for resolution improvement –Predictive method less expensive than phase

Detailed Properties of Predictive Method Two components to the errors –Systematic component –Random component A B C D E A B C D E

Effect of Match Window Size Systematic bias –No significant change above 20 x 20 pixels –Bias is therefore inherent in the method used Random component –Depends significantly on window size for small windows –When random component has approximately same magnitude as the bias, it stops improving –No further improvement above about 100 x 100 pixels –Above this, the error has distinct double peak –Random component is therefore limited by systematic bias

Summary of Predictive Registration Very fast –Requires only a single pass through the image –(2 passes if not already registered to nearest pixel) Accuracy is 4 - 5% of pixel –This is between pairs of images –May be improved by enforcing consistency Optimum match window size is about 100 x 100 pixels Robust to moderate levels of noise Relatively insensitive to aliasing Suitable for images captured in identical conditions –Sensitive to contrast and brightness

Overall Summary Described resolution improvement –Reconstruction steps: registration, resampling, inverse filtering –Examined some of the preconditions: aliasing –Discussed some of the limitations: lens and camera blurring Presented results from 1D –Super-resolution of bar code images Presented results from 2D registration –Comparison of registration methods –Description of a new fast method Predictive Interpolation method

References “Super-resolution of bar-codes”, D.G. Bailey, Journal of Electronic Imaging, vol 10 (1), pp (2001) “Predictive Interpolation for Registration”, D.G. Bailey, Proceedings of Image and Vision Computing Conference NZ, pp (November 2000) “Image Registration Methods for Resolution Improvement”, D.G. Bailey and T.H. Lill, Proceedings of Image and Vision Computing NZ, pp (August 1999) “Super-Resolution of Bar Codes”, D.G. Bailey, SPIE Proceedings, Vol 3521 Machine Vision Systems for Inspection and Metrology VII, Boston, pp (November 1998)