1. SURF applied in Panorama Image Stitching
2010/1/10
SURF applied in Panorama Image Stitching
Keywords: panorama, SURF, stitching, multi-band blending, LM, bundle adjustment

2. Outline Introduction Modified SURF
Comparison of SIFT and modified SURF
Experiments and results
Time cost test
Conclusion and future works
Reference

3. Introduction
Stitching multiple images is a popular method of effectively increasing the field of view of a camera, by allowing several views of a scene to be combined into a single view.
Stitching includes two main parts: image matching and image blending.
find the motion relationship between two images or several images

4. Introduction-2
For matching, there are two ways: direct method and feature detection method
Direct method: is inconvenient because it always needs a high quality image.
feature detection method: such as SURF and SIFT.
For blending: weight averaged linear blending, multi-band blending…etc
multi-band blending: good performance between quality of result and time cost

5. Introduction-3
This paper a panorama image stitching process which combines an image matching system: Modified SURF + multi-band blending.
1. find the SURF features from the image(use KNN and RANSAC)
2. estimate the homography matrix according to LM (Levenberg-Marquardt) method,
3. adjust the coordinate of images
4. blend the images by multi-blending to remove the stitch seam and illumination discrepancy

6. Modified SURF SURF method:
relying on integral images for image convolutions
building on the strengths of the leading existing detectors and descriptors (using a Hessian matrix-based measure for the detector, and a distribution-based descriptor)
simplifying these methods to the essential

7. Modified SURF-2
Integral images allow the computation of rectangular box filters in near constant time
This paper used KNN to find the nearest neighbor with setting k to 2
RANSAC is used to estimate a model for finding the minimize error matches set, which can maintain the correct matches by comparing the distance of the closest neighbor to that of second-closest neighbor

8. Comparison of SIFT and modified SURF
After extracting invariant scale features, we got potential feature matches by using k-nearest neighbor method, and then remove the mismatches with RANSAC algorithm.
→ modified SURF
Though SIFT get more matches than Modified SURF, Modified SURF is much faster than SIFT

9. Comparison of SIFT and modified SURF-2
Figure 1 and Fig. 2 give the match results of Modified SURF and SIFT in scale and illumination change respectively:
win
win

10. Comparison of SIFT and modified SURF-3
modified SURF was not better than SIFT in rotation, Modified SURF is as robust as SIFT in other performance.

11. Comparison of SIFT and modified SURF-4
Summary:
SURF describes image faster than SIFT by 3 times
SURF is not as well as SIFT on invariance to illumination change and viewpoint change

12. Adjustment and blending
bundle adjustment: to transform the images into a same coordinate or computing surface
1. Choose one of the images to be reference surface
2. transform each of other images to the reference surface
usual choice for compositing larger panoramas is to use a cylindrical or spherical projection.

13. Adjustment and blending-2
Transformation: compute the homography and optimize the parameters of the matrix
1. find out the best neighbor image for each image
2. calculate the distance between the two neighbor images
3. minimize the distance value to adjust the matrix between the neighbor images (use LM)

14. Adjustment and blending-3
Levenberg-Marquardt (LM): 用來解決nonlinear function的least square問題, using nonlinear minimum square evaluation。
to minimize the transfer error, which is calculated as equation:
Euclidian distance
Is correspondent with points X’
Homography matrix

15. Image Blending algorithm
goal : is to produce a resulting image where no transition can be seen between the original source images.
Linear method: may results in ghosting artifacts, blurring.
But Linear blending method is fast and can be a good compromise between quality and speed if you are not too demanding on quality

16. Image Blending algorithm-2
multi-band blending (or called by pyramid blending): effective for image stitching without blurring and ghosting artifacts.
It will produce much better results than the "Linear" mode.
Multiband blending scheme ensures smooth transitions between images despite illumination differences

17. Image Blending algorithm-3
compare two methods:
Refernce:

18. Image Blending algorithm-4
Multi-band Blending:
compare to linear blending，multi-band blending can make image more clear in detail. In the paper, author use 2-band.
將影像分成兩個band：低頻和高頻
低頻影像用較大的weighting(使用較大σ 之Gaussian function)，高頻影像用較小的weighting 方式(使用較小σ 之Gaussian function) 。兩個band 各自做完blending 後，再合起來。

19. Image Blending algorithm-5
The Laplacian pyramid of the final image is formed as equation:
Pyramid blending gradually blends the lower frequencies of the images while maintaining a sharper transition for the higher frequencies
Yk is the kth level of Laplacian pyramid decomposition for the final combination result
Mk is the kth level of Gaussian pyramid
decomposition of the image mas
where X1,k and X2,k are the kth level of Laplacian pyramid decomposition

20. Experiments and results
Environment: studio 2008 C++ with OpenCV library
Flow diagram has tow parts: matching and blending
The connection of the two parts is the correspondence pairs.

21. Experiments and results-2
Flow of matching: the goal of which is to find the largest feature points good to transformation
square Euclidean distance ratio between neighbors is calculated
Detect feature points
estimate a model of consensus set that minimizes matching error
Output for blending as itsinput

22. Experiments and results-3
Flow of blending:
H can be estimated based on those correspondence pairs
the images has been
transformed into the corresponding image in a same coordinate system by the H matrix
Correct some stitching error of color and illumination.

23. Experiments and results-4
Experiments consist of two parts: panorama quality (stitching) test and time cost test.
A good stitching program should make panorama seamless and clear and be fast for using in various application such as real time processing.

24. Experiments and results-5
Stitching test: there are three seams in Fig.4 1 3 2
Fig. 4. Panorama with obvious seam before blending processing
Fig. 5. Panorama with seamless after blending processing of Fig. 4

25. Experiments and results-6
Next, we did an experiment with large data set. In this experiment, we use 16 images of Camp dataset
We will see that the present stitching process can show its good performance for large image dataset

26. Experiments and results-7
Fig. 6. Panorama stitched 16 images based on SIFT.
Fig.7. Panorama stitched 16 images based on modified SURF

27. Time cost test
The present system is faster than SIFT demo as shown in Fig 8 and Fig 9, due to using fast modified SURF.
The present system is almost 4 times (3.56~4.46) faster than SIFT demo
Fig. 8. Time cost– number of images

28. Time cost test-2
Each experiment the time will change a little because of the CPU and memory, the reasonable experiment time is needed
Fig 9 shows the time cost when the size of images that used are different

29. Conclusion and future works
The present panorama image stitching process has a good performance.
Due to Modified SURF, high-resolution panorama can be created in case there are some changes of illumination, color, blur and et cetera, and processed fast.

30. Conclusion and future works-2
Reasons for good performance:
K-NN and RANSAC improves the repeatability of matching.
Bundle adjustment and multi-band blending make the panorama seamless.
LM is used to estimate the homography, which makes the transformation more accurate

31. Conclusion and future works-3
SURF is poor at
handling viewpoint change
handling illumination change
Plus, the present system shows its defects when there are some noise images that are not neighbored
removing the noise before stitching processing

32. Reference
Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool. SURF: Speeded Up Robust Features. Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp , 2008.
Luo Juan and Oubong Gwun. A Comparison of SIFT, PCA-SIFT and SURF. International Journal of Image Processing (IJIP), Volume (3): Issue (4) 2009.