1. Spatial and Spectral Evaluation of Image Fusion Methods Sascha Klonus Manfred Ehlers Institute for Geoinformatics and Remote Sensing University of Osnabrück

2. Content Introduction Image Fusion Test Site Fusion Results Color Distortions Evaluation Methods and Results Ehlers Fusion Conclusions and Future Work

3. Remote sensors have different spatial resolution for panchromatic and multispectral imagery The ratios vary between 1:2 and 1:5 For multisensor fusion the ratios can exceed 1:30 (e.g. Ikonos/Landsat) Data Fusion: Why is it Necessary?

4. Objectives of Image Fusion Sharpen images Improve geometric corrections Provide stereo-viewing capabilities Enhance certain features Complement data sets Detect changes Substitute missing information Replace defective data Pohl & van Genderen (1998)

5. Meaning of Pan-Sharpening SpatialSpectral + panchromatic & high geometric resolution multi-/hyperspectral image & low geometric resolution multi-/hyperspectral & high geometric resolution

6. Fusion Methods Color Transformations  Modified IHS Transformation Statistical Methods  Principal Component Merge Numerical Methods  Brovey  CN Spectral Sharpening  Gram-Schmidt Spectral Sharpening  Wavelet based Fusion Combined Methods  Ehlers Fusion

7. Test Site

8. Original Data Quickbird Multispectral image (2004-09-04) Quickbird Panchromatic image (2004-09-04) Formosat Multispectral image (2004-01-30) Ikonos Multispectral image (2005-08-03)

9. Single Sensor Fusion: Quickbird Quickbird Multispectral image Fused with BroveyFused with CN Spectral SharpeningFused with Ehlers Fused with Wavelet Fused with Gram-Schmidt Fused with PC Fused with modified IHS

10. Multisensor Fusion: Ikonos Ikonos Multispectral image Fused with BroveyFused with CN Spectral Sharpening Fused with Ehlers Fused with modified IHS Fused with PCFused with Gram-Schmidt Fused with Wavelet

11. Multisensor Fusion: Formosat Formosat Multispectral image Fused with Brovey Fused with CN Spectral Sharpening Fused with Ehlers Fused with modified IHSFused with PCFused with Gram-Schmidt Fused with Wavelet

12. Panchromatic band has a different spectral sensitivity Multisensoral differences (e.g. Ikonos and SPOT merge) Multitemporal (seasonal) changes between pan and ms image data Fusion Problem: Color Distortion  Inconsistent panchromatic information is fused into the multispectral bands

13. Spectral Comparison Methods (1) s = standard deviation org = Original image fused = Fused image x = Mean RMSE Correlation coefficients Visual (Structure and Colour Preservation)

14. Results RMSE QuickbirdIkonosFormosat Mod. IHS7.182211.07141.7792 PC42.505823.897923.3095 Brovey222.1732143.830314.3147 CN-Sharpening46.7389123.503870.4803 Gram-Schmidt5.68835.14250.4078 Wavelet3.19152.53790.0290 Ehlers1.00284.36450.4201

15. Results Correlation Coefficients QuickbirdIkonosFormosat Mod. IHS0.87370.57310.5892 PC0.88110.65120.6199 Brovey0.86110.60200.5677 CN-Sharpening0.86110.6020-0.0546 Gram-Schmidt0.86900.67740.6110 Wavelet0.96980.97190.9720 Ehlers0.95100.80940.9324

16. Spectral Comparison Methods (2) Per Pixel Deviation Degrade Degraded to ground resolution of original image (Formosat = 8m) Original multispectral image (Formosat 8m) Band 12.56 Band 22.92 Band 33.49 Band 43.35 Result: Vector containing the deviation per pixel Fused image (Formosat 2m)

17. Mean Per Pixel Deviation QuickbirdIkonosFormosat IHS27.2842.9511.93 PC48.8053.2722.52 Brovey209.15136.6219.48 CN-Sharpening48.73117.6170.71 Gram-Schmidt30.3545.2911.99 Wavelet7.2013.473.04 Ehlers17.2825.864.29

18. Spatial Comparison Methods (1) Edge Detection - - - Band 191.16 % Band 292.10 % Band 392.64 % Mean91.96 %

19. Results Edge Detection QuickbirdIkonosFormosat Mod. IHS92.71 %92.44 %95.54 % PC95.10 %93.28 %93.44 % Brovey94.69 %95.16 %97.87 % CN-Sharpening94.69 %95.16 %90.69 % Gram-Schmidt95.02 %95.53 %97.82 % Wavelet85.00 %83.82 %84.81 % Ehlers91.85 %90.35 %94.40 %

20. Spatial Comparison Methods (2) Highpass Filtering Correlation Band 10.8012 Band 20.7820 Band 30.7912 Mean0.7918

21. Highpass Correlation Results QuickbirdIkonosFormosat Mod. IHS0.93360.91490.9420 PC0.99000.00210.8073 Brovey0.97150.97650.9895 CN-Sharpening0.97140.9764-0.0170 Gram-Schmidt0.98570.98790.9652 Wavelet0.39760.36270.3799 Ehlers0.89970.86890.9349

22. FFT Filter Based Data Fusion (Ehlers Fusion) Panchromatic Image Multispectral Image RGBRGB Basis: IHS Transform and Filtering in the Fourier Domain FFT Fourier Spectrum FFT Fourier Spectrum HPF Pan HP LPF I LP IHSIHS R‘ G‘ B‘ IHS -1 I LP +Pan HP H S FFT -1

23. Panchromatic image and its spectrum Original panchromatic image Panchromatic Spectrum

24. Filtersetting effects Frequency Intensity Cut-off Frequency fnfn Filtered Panchromatic Spectrum

25. Effects in the spatial domain Filtered panchromatic imageFused image

26. Filtersetting effects Frequency Intensity Cut-off Frequency fnfn Filtered Panchromatic Spectrum

27. Effects in the spatial domain Filtered panchromatic imageFused image

28. Filtersetting effects Filtered Panchromatic Spectrum Frequency Intensity Cut-off Frequency fnfn

29. Effects in the spatial domain Filtered panchromatic imageFused image

30. Results Ehlers Fusion shows the best overall results in all images It works also if the panchromatic Information does not match the spectral sensitivity of the merged bands (multitemporal and multisensoral fusion) Its performance is superior to standard fusion techniques (IHS, Brovey Transform, PC Merge) Wavelet preserves the spectral characteristics at the cost of spatial improvement Ehlers Fusion is integrated in a commercial image processing system (Erdas Imagine 9.1)

31. Future Work Fusion of radar- and optical Data Development of one method to evaluate the spatial and spectral quality of an fused image Comparison with the algorithm of Zhang (PCI Geomatica) Research on automation for filter design

32. Thanks for your Attention Questions???

33. Ehlers Fusion Program

38. Multispectral image and its spectrum Original multispectral intensity Multispectral intensity spectrum

39. Filtersetting effects Filtered multispectral spectrum Frequency Intensity Cut-off Frequency fnfn

40. Filtersetting effects Filtered multispectral spectrum Frequency Intensity Cut-off Frequency fnfn

41. Filtersetting effects Filtered multispectral spectrum Frequency Intensity Cut-off Frequency fnfn