1. ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina Department of Computer Science Ioannina, Greece

2. ObjectiveObjective – –Reconstruct a high-resolution image from a sequence of low-resolution images. Improve spatial resolution. Constraints on low-resolution imagesConstraints on low-resolution images – –Motion – –Rotation – –Blurring – –Subsampling – –Additive noise MOTIVATION

3. MAP scheme for image super-resolution. Registration in two parts – SIFT ( Scale- Invariant Feature Transform ) –At first, the LR images are registered by establishing correspondences between robust SIFT (Scale- Invariant Feature Transform) features. – –In the second step, the estimation of the registration parameters is fine tuned along with the estimation of the HR image. Mutual Information Criterion:Mutual Information Criterion: maximize the mutual information between HR image and each of the upscaled LR images. APPROACH

4. Let the high-resolution image where The set of LR images is described as We consider p LR images each of size FORMULATION MODEL

5. Observation model:Observation model: – –All images are ordered lexicographically – – represents zero-mean additive Gaussian noise, – – is the degradation matrix, performing the operations of: motion blur down-sampling FORMULATION MODEL (cont.)

6. The Gaussian prior for the HR image is: – – is the Laplacian of the image z – – controls the precision and the shape of the distribution The likelihood of the LR images is Gaussian: MAP ESTIMATOR

7. MAP approachMAP approach – –Maximize – –Which leads to a MAP functional to be minimized with respect to HR image z and the transformation parameters s: Use gradient descent methodUse gradient descent method – –The update equation is given by: where ε n is the step size at the n-th iteration. MAP ESTIMATOR (cont.)

8. Objective:Objective: independently detect corresponding keypoints in scaled versions of the same image. Idea:Idea: Given a keypoint in two images, determine if the surrounding neighborhoods contain the same structure up to scale. SIFT features are invariant to: – –Image scale and rotation – –Affine transformations – –Changes in illumination and noise [ D. G. Lowe. "Distinctive image features from scale invariant keypoints.”International Journal of Computer Vision 60 (2), pp. 91-110, 2004. ] SCALE INVARIANT FEATUTE TRANSFORM - SIFT

9. Basics:Basics: the mutual information is maximized when the two images are correctly registered. The mutual information between two images A and B is: – –H(A) and H(B) are the marginal entropies of the random variables A and B. – –H(A,B) is the joint entropy. MUTUAL INFORMATION CRITERION

10. Normalized Mutual Information:Normalized Mutual Information: – –Robust measure in order to provide invariance to the overlapping areas between the two images. Problem:Problem: – –If mutual information is not initialized close to the optimal solution it is trapped by local maxima. Good initialization is important. Solution:Solution: – SIFT –Initialization using SIFT descriptors. MUTUAL INFORMATION CRITERION (cont.)

11. Estimation of registration parameters in two steps. – SIFT –First step, LR images are registered by employing SIFT features. Minimization of mean square error between the locations of features between the reference image and the LR images. Provides good initialization. IMAGE REGISTRATION

12. – –Second step, the estimation of the registration parameters is fine-tuned along with the estimation of the HR image, by maximization of mutual information criterion. Iterative scheme. Contribution: – –The registration accuracy is improved at each iteration step. – –Refinement of the mutual information registration. IMAGE REGISTRATION (cont.)

13. Synthetic data sets. LR images were created by rotating, translating, blurring, down-sampling and degrading by noise. –Translation: –Translation: uniformly selected in [-3, 3] (in pixels) –Rotation: –Rotation: uniformly selected in [-5, 5] (in degrees) –Down-sampling factor: –Down-sampling factor: 2 (4 pixels to 1) –Blurring: –Blurring: 5x5 Gaussian kernel, standard deviation of 1 –Additive noise: –Additive noise: AWGN to obtain SNR of 30 dB and 20 dB EXPERIMENTAL PARAMETERS

14. First estimate of the HR image – –Bicubic interpolation Total number of realizations for each case:Total number of realizations for each case: 10 Convergence:Convergence: or 70 iterations reached. Quantitative evaluation:Quantitative evaluation: peak signal to noise ratio EXPERIMENTAL PARAMETERS (cont.)

15. COMPARE METHODS

16. Books (PSNR = 26.06 dB) 4 LR images used EXPERIMETAL RESULTS LR image Reconstructed HR image

17. Front page (PSNR = 26.14 dB) 6 LR images used EXPERIMETAL RESULTS (cont.) LR image Reconstructed HR image

18. Car (PSNR = 28.13 dB) 5 LR images used EXPERIMETAL RESULTS (cont.) LR image Reconstructed HR image

19. Eye chart (PSNR = 27.33 dB) 4 LR images used EXPERIMETAL RESULTS (cont.) LR image Reconstructed HR image

20. EXPERIMETAL RESULTS (cont.) Statistics for the compared SR methods +1.5 dB +1.5 dB on average better results than SIFT.

21. Hybrid registration approach – –SIFT-based image registration combined with the maximization of mutual information. – –High precision registration High accuracy super-resolved image. – –Improvement is 1.5 dB on average higher for both 30 dB and 20 dB. Proposed algorithm converges faster than the standard solution. CONCLUTIONS

22. QUESTIONS?

23. THANK YOU ALL FOR YOUR PARTICIPATION AND PATIENCE!